[Oisf-users] Inconsistent packet dropped behaviour with the same config on several nodes

Fri Oct 19 11:04:02 UTC 2018

Dear Peter,

I'm sorry for the late reply.
I have started suricata without any rules on the problematic node2, and
indeed the drops were almost 0.
By default we have around 20k rules loaded.
On the other hand, with that little traffic we would still expect having
very little/no drops.

The other node (node1) has exactly same set of rules loaded.

Thank you a lot for you help.

Best,
magmi

On Fri, 14 Sep 2018 at 11:13, Peter Manev <petermanev at gmail.com> wrote:

> On Thu, Sep 13, 2018 at 8:48 AM Magmi A <magmi.sec at gmail.com> wrote:
> >
> > Hi Peter,
> >
> > Yes, of course.
> > Here you have stats.log for Node2:
> >
> > Date:9/13/2018--06:37:42(uptime:6d,17h05m59s)
> >
> ------------------------------------------------------------------------------------
> > Counter |TMName |Value
> >
> ------------------------------------------------------------------------------------
> > capture.kernel_packets |Total |135374454
> > capture.kernel_drops |Total |11430384
> > decoder.pkts |Total |123946335
> > decoder.bytes |Total |122943123694
> > decoder.ipv4 |Total |123767936
> > decoder.ipv6 |Total |5036
> > decoder.ethernet |Total |123946335
> > decoder.tcp |Total |120082602
> > decoder.udp |Total |683567
> > decoder.icmpv4 |Total |114883
> > decoder.icmpv6 |Total |189
> > decoder.tered |Total |5
> > decoder.avg_pkt_size |Total |991
> > decoder.max_pkt_size |Total |1514
> > flow.tcp |Total |780742
> > flow.udp |Total |305951
> > flow.icmpv6 |Total |162
> > tcp.sessions |Total |727356
> > tcp.syn |Total |771112
> > tcp.synack |Total |720764
> > tcp.rst |Total |549359
> > tcp.stream_depth_reached |Total |454
> > tcp.reassembly_gap |Total |7722
> > tcp.overlap |Total |483624
> > detect.alert |Total |4
> > app_layer.flow.http |Total |108080
> > app_layer.tx.http |Total |262748
> > app_layer.flow.smtp |Total |7
> > app_layer.tx.smtp |Total |7
> > app_layer.flow.tls |Total |6612
> > app_layer.flow.ssh |Total |13
> > app_layer.flow.smb |Total |55361
> > app_layer.flow.dcerpc_tcp |Total |202204
> > app_layer.flow.dns_tcp |Total |10
> > app_layer.tx.dns_tcp |Total |10
> > app_layer.flow.failed_tcp |Total |274419
> > app_layer.flow.dcerpc_udp |Total |4
> > app_layer.flow.dns_udp |Total |139325
> > app_layer.tx.dns_udp |Total |239577
> > app_layer.flow.failed_udp |Total |166622
> > flow_mgr.closed_pruned |Total |684943
> > flow_mgr.new_pruned |Total |290500
> > flow_mgr.est_pruned |Total |110950
> > flow.spare |Total |10000
> > flow.tcp_reuse |Total |6055
> > flow_mgr.flows_checked |Total |12
> > flow_mgr.flows_notimeout |Total |5
> > flow_mgr.flows_timeout |Total |7
> > flow_mgr.flows_timeout_inuse |Total |1
> > flow_mgr.flows_removed |Total |6
> > flow_mgr.rows_checked |Total |65536
> > flow_mgr.rows_skipped |Total |65522
> > flow_mgr.rows_empty |Total |2
> > flow_mgr.rows_maxlen |Total |1
> > tcp.memuse |Total |4587520
> > tcp.reassembly_memuse |Total |6650304
> > dns.memuse |Total |16386
> > http.memuse |Total |12142178
> > flow.memuse |Total |7207360
> >
>
> Can you do a zero test and see if you run it for  a bit with 0 rules
> loaded ( -S /dev/null )- if you are going to have the same amount of
> drops percentage wise?
>
>
>
> > And just for a reference stats.log for Node1:
> >
> > Date:9/13/2018--06:35:39(uptime:6d,16h39m54s)
> >
> ------------------------------------------------------------------------------------
> > Counter |TMName |Value
> >
> ------------------------------------------------------------------------------------
> > capture.kernel_packets |Total |3577965800
> > capture.kernel_drops |Total |3416155
> > decoder.pkts |Total |3574589712
> > decoder.bytes |Total |3536139875210
> > decoder.invalid |Total |10070
> > decoder.ipv4 |Total |3571132083
> > decoder.ipv6 |Total |143756
> > decoder.ethernet |Total |3574589712
> > decoder.tcp |Total |3522243739
> > decoder.udp |Total |34827953
> > decoder.icmpv4 |Total |963831
> > decoder.icmpv6 |Total |33551
> > decoder.teredo |Total |1399
> > decoder.avg_pkt_size |Total |989
> > decoder.max_pkt_size |Total |1534
> > flow.tcp |Total |1144524
> > flow.udp |Total |202960
> > flow.icmpv6 |Total |439
> > decoder.ipv4.trunc_pkt |Total |10070
> > tcp.sessions |Total |341278
> > tcp.ssn_memcap_drop |Total |4446979
> > tcp.pseudo |Total |84
> > tcp.invalid_checksum |Total |4
> > tcp.syn |Total |6653717
> > tcp.synack |Total |2572744
> > tcp.rst |Total |1857715
> > tcp.segment_memcap_drop |Total |10
> > tcp.stream_depth_reached |Total |303
> > tcp.reassembly_gap |Total |95648
> > tcp.overlap |Total |3889304
> > tcp.insert_data_normal_fail |Total |3314483
> > detect.alert |Total |518
> > app_layer.flow.http |Total |34820
> > app_layer.tx.http |Total |60759
> > app_layer.flow.ftp |Total |20
> > app_layer.flow.smtp |Total |140
> > app_layer.tx.smtp |Total |177
> > app_layer.flow.tls |Total |43356
> > app_layer.flow.smb |Total |3430
> > app_layer.flow.dcerpc_tcp |Total |8894
> > app_layer.flow.dns_tcp |Total |48
> > app_layer.tx.dns_tcp |Total |46
> > app_layer.flow.failed_tcp |Total |107518
> > app_layer.flow.dcerpc_udp |Total |5
> > app_layer.flow.dns_udp |Total |114888
> > app_layer.tx.dns_udp |Total |482904
> > app_layer.flow.failed_udp |Total |88067
> > flow_mgr.closed_pruned |Total |259368
> > flow_mgr.new_pruned |Total |981024
> > flow_mgr.est_pruned |Total |107531
> > flow.spare |Total |10000
> > flow.tcp_reuse |Total |29932
> > flow_mgr.rows_checked |Total |65536
> > flow_mgr.rows_skipped |Total |65536
> > tcp.memuse |Total |4587520
> > tcp.reassembly_memuse |Total |655360
> > dns.memcap_global |Total |1086836
> > flow.memuse |Total |7074304
> >
> > Thank you for any suggestions.
> >
> > Best,
> > magmi
> >
> > On Wed, 12 Sep 2018 at 16:16, Peter Manev <petermanev at gmail.com> wrote:
> >>
> >> On Wed, Sep 12, 2018 at 10:57 AM Magmi A <magmi.sec at gmail.com> wrote:
> >> >
> >> >
> >> >> > * Node1 receives ~ 500Mbps of traffic (it's 1Gbps interface), and
> gets in average 1-2% kernel packet dropped
> >> >> > while
> >> >> > * Node2 receives ~ 500kbps of traffic and gets in average 10%
> kernel packet dropped
> >> >>
> >> >> What is different between node 1 and node 2 ? (same config/same
> >> >> suricata/same HW/same rules...?)]
> >> >
> >> >
> >> > The nodes have the same HW, run the same config/ suricata version,
> have the same set of rules.
> >> >
> >> > The only difference is that they are exposed to different sources of
> traffic.
> >> > From Wireshark analysis the protocol hierarchies for both cases seem
> similar - there is no spectacular difference.
> >> >
> >> > So really the only difference is the captured traffic itself (MACs,
> IPs, partly protocols, data etc).
> >> >
> >> > That is why we have such a problem how to approach the problem and
> troubleshoot it.
> >>
> >>
> >> Can you share full update of the latest stats.log ?
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
>
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20181019/5d6d7ebf/attachment-0001.html>