[Oisf-users] troubleshooting packet loss

Peter Manev petermanev at gmail.com
Thu Mar 31 13:24:44 UTC 2016 

On Thu, Mar 31, 2016 at 3:16 PM, Yasha Zislin <coolyasha at hotmail.com> wrote:
> detect:
>   - profile: medium
>   - custom-values:
>       toclient-groups: 3
>       toserver-groups: 25
>
>   - sgh-mpm-context: auto
>   - inspection-recursion-limit: 3000

You should follow the standard here -
https://github.com/inliniac/suricata/blob/dev-detect-grouping-v193/suricata.yaml.in#L594
Notice that you have "-" in "- profile: medium".... it should be
"profile: custom" ...etc.

Then just increase the groups numbers to 500/600 for example and use
"sgh-mpm-context: full"

Make sure it reflects that correctly while loading in suricata.log
using the "-v" switch.

>
>
>> Date: Thu, 31 Mar 2016 15:03:07 +0200
>
>> Subject: Re: [Oisf-users] troubleshooting packet loss
>> From: petermanev at gmail.com
>> To: coolyasha at hotmail.com
>> CC: oisf-users at lists.openinfosecfoundation.org
>>
>> On Thu, Mar 31, 2016 at 3:01 PM, Yasha Zislin <coolyasha at hotmail.com>
>> wrote:
>> >
>> > Peter,
>> >
>> > max_packets is set to 65000.
>> >
>> > So I've set my detect-engine to profile medium and default groups like
>> > in your documentation.
>>
>> Can you please paste that section here?
>>
>> > Packet loss from the start is 96%
>> >
>> > ________________________________
>> > CC: oisf-users at lists.openinfosecfoundation.org
>> > From: petermanev at gmail.com
>> > Subject: Re: [Oisf-users] troubleshooting packet loss
>> > Date: Mon, 28 Mar 2016 19:40:50 +0200
>> >
>> > To: coolyasha at hotmail.com
>> >
>> >
>> > On 28 mars 2016, at 19:19, Yasha Zislin <coolyasha at hotmail.com> wrote:
>> >
>> > detect-engine:
>> > - profile: custom
>> > - custom-values:
>> > toclient-src-groups: 200
>> > toclient-dst-groups: 200
>> > toclient-sp-groups: 200
>> > toclient-dp-groups: 300
>> > toserver-src-groups: 200
>> > toserver-dst-groups: 400
>> > toserver-sp-groups: 200
>> > toserver-dp-groups: 250
>> > - sgh-mpm-context: auto
>> > - inspection-recursion-limit: 3000
>> > # When rule-reload is enabled, sending a USR2 signal to the Suricata
>> > process
>> > # will trigger a live rule reload. Experimental feature, use with care.
>> > - rule-reload: true
>> >
>> >
>> > Can you please try adjusting the settings (including the yaml section
>> > itself) as per the link and recommendations I have mentioned in my previous
>> > mail.
>> > The above quoted part of your suricata.yaml section do not follow the
>> > dev-detect-grouping standard/change.
>> >
>> > For pf_ring, I am using 6.3.0. On this specific sensor, I only have one
>> > interface.
>> > Here is suricata.yaml config.
>> > - interface: bond0
>> > # Number of receive threads (>1 will enable experimental flow pinned
>> > # runmode)
>> > threads: 2
>> >
>> > # Default clusterid. PF_RING will load balance packets based on flow.
>> > # All threads/processes that will participate need to have the same
>> > # clusterid.
>> > cluster-id: 99
>> >
>> > # Default PF_RING cluster type. PF_RING can load balance per flow or per
>> > hash.
>> > # This is only supported in versions of PF_RING > 4.1.1.
>> > cluster-type: cluster_flow
>> >
>> >
>> > What is the max pending packets value you have in yaml ?
>> >
>> > Thanks
>> >
>> > Here are pf_ring settings that I use in startup script.
>> >
>> > ethtool -K bond0 rx off
>> > ethtool -K bond0 tx off
>> > ethtool -K bond0 gso off
>> > ethtool -K bond0 gro off
>> >
>> > ethtool -C bond0 rx-usecs 500
>> > ethtool -G bond0 rx 4078
>> >
>> > ifconfig bond0 promisc
>> >
>> > rmmod pf_ring
>> > modprobe pf_ring transparent_mode=0 min_num_slots=65576
>> > enable_tx_capture=0
>> >
>> > ________________________________
>> > CC: oisf-users at lists.openinfosecfoundation.org
>> > From: petermanev at gmail.com
>> > Subject: Re: [Oisf-users] troubleshooting packet loss
>> > Date: Mon, 28 Mar 2016 18:47:56 +0200
>> > To: coolyasha at hotmail.com
>> >
>> >
>> > On 28 mars 2016, at 14:30, Yasha Zislin <coolyasha at hotmail.com> wrote:
>> >
>> > I have about 2 million packets traversing per minute over 10 gig
>> >
>> > pipe. Averaging about 500mbits in traffic.
>> >
>> > Traffic is mostly LDAP, NTP, Syslog. Not really much of HTTP/S.
>> >
>> > That packet loss starts instantly. In a few minutes, you would see it.
>> > After leaving it running over the weekend, I have 19% packet loss.
>> >
>> > BTW, I am using dev v190 branch for this already.
>> >
>> >
>> > What does your "detect:" section look like in suricata.yaml?
>> >
>> > You can use the section here as a reference -
>> >
>> > https://github.com/inliniac/suricata/blob/dev-detect-grouping-v193/suricata.yaml.in#L594
>> > (Please note the naming and spacing)
>> >
>> >
>> > Are you pfring buffers full ? (Or how are they configured )
>> > How does the config section look like for pfring for the two interfaces
>> > you have?
>> >
>> >
>> >
>> > Thanks.
>> >
>> > > Date: Sun, 27 Mar 2016 12:00:22 +0200
>> > > Subject: Re: [Oisf-users] troubleshooting packet loss
>> > > From: petermanev at gmail.com
>> > > To: coolyasha at hotmail.com
>> > > CC: oisf-users at lists.openinfosecfoundation.org
>> > >
>> > > On Thu, Mar 24, 2016 at 7:02 PM, Yasha Zislin <coolyasha at hotmail.com>
>> > > wrote:
>> > > > I am trying to figure out where the packet loss is coming from on
>> > > > one of my
>> > > > Suricata 3.0 sensor.
>> > > > The only thing that I see weird from stats.log is that
>> > > > tpc.stream_depth_reached and tcp.reassembly_gap is somewhat high.
>> > > > I am using latest PF_RING and monitoring one interface with 4
>> > > > threads.
>> > > > 4 logical CPUs with 16 gigs of RAM. 66% of RAM is used.
>> > >
>> > > What traffic speeds are those on? How many rules do you load?
>> > >
>> > > On the first interface there is 6.5% loss on the second 3.67% - over
>> > > what period of time was that?
>> > >
>> > > >
>> > > > Here is stats.log info.
>> > > >
>> > > > Thank you
>> > > >
>> > > > capture.kernel_packets | RxPFRbond01 | 34118172
>> > > > capture.kernel_drops | RxPFRbond01 | 2240130
>> > > > decoder.pkts | RxPFRbond01 | 34125944
>> > > > decoder.bytes | RxPFRbond01 | 26624108366
>> > > > decoder.invalid | RxPFRbond01 | 0
>> > > > decoder.ipv4 | RxPFRbond01 | 34707873
>> > > > decoder.ipv6 | RxPFRbond01 | 570
>> > > > decoder.ethernet | RxPFRbond01 | 34125944
>> > > > decoder.raw | RxPFRbond01 | 0
>> > > > decoder.null | RxPFRbond01 | 0
>> > > > decoder.sll | RxPFRbond01 | 0
>> > > > decoder.tcp | RxPFRbond01 | 23715873
>> > > > decoder.udp | RxPFRbond01 | 9702569
>> > > > decoder.sctp | RxPFRbond01 | 0
>> > > > decoder.icmpv4 | RxPFRbond01 | 98456
>> > > > decoder.icmpv6 | RxPFRbond01 | 0
>> > > > decoder.ppp | RxPFRbond01 | 0
>> > > > decoder.pppoe | RxPFRbond01 | 0
>> > > > decoder.gre | RxPFRbond01 | 0
>> > > > decoder.vlan | RxPFRbond01 | 0
>> > > > decoder.vlan_qinq | RxPFRbond01 | 0
>> > > > decoder.teredo | RxPFRbond01 | 570
>> > > > decoder.ipv4_in_ipv6 | RxPFRbond01 | 0
>> > > > decoder.ipv6_in_ipv6 | RxPFRbond01 | 0
>> > > > decoder.mpls | RxPFRbond01 | 0
>> > > > decoder.avg_pkt_size | RxPFRbond01 | 780
>> > > > decoder.max_pkt_size | RxPFRbond01 | 1514
>> > > > decoder.erspan | RxPFRbond01 | 0
>> > > > flow.memcap | RxPFRbond01 | 0
>> > > > defrag.ipv4.fragments | RxPFRbond01 | 1190975
>> > > > defrag.ipv4.reassembled | RxPFRbond01 | 592903
>> > > > defrag.ipv4.timeouts | RxPFRbond01 | 0
>> > > > defrag.ipv6.fragments | RxPFRbond01 | 0
>> > > > defrag.ipv6.reassembled | RxPFRbond01 | 0
>> > > > defrag.ipv6.timeouts | RxPFRbond01 | 0
>> > > > defrag.max_frag_hits | RxPFRbond01 | 0
>> > > > tcp.sessions | RxPFRbond01 | 169101
>> > > > tcp.ssn_memcap_drop | RxPFRbond01 | 0
>> > > > tcp.pseudo | RxPFRbond01 | 77497
>> > > > tcp.pseudo_failed | RxPFRbond01 | 0
>> > > > tcp.invalid_checksum | RxPFRbond01 | 0
>> > > > tcp.no_flow | RxPFRbond01 | 0
>> > > > tcp.syn | RxPFRbond01 | 180407
>> > > > tcp.synack | RxPFRbond01 | 146913
>> > > > tcp.rst | RxPFRbond01 | 138896
>> > > > tcp.segment_memcap_drop | RxPFRbond01 | 0
>> > > > tcp.stream_depth_reached | RxPFRbond01 | 107
>> > > > tcp.reassembly_gap | RxPFRbond01 | 6765
>> > > > detect.alert | RxPFRbond01 | 3426
>> > > > capture.kernel_packets | RxPFRbond02 | 33927252
>> > > > capture.kernel_drops | RxPFRbond02 | 1246692
>> > > > decoder.pkts | RxPFRbond02 | 33932611
>> > > > decoder.bytes | RxPFRbond02 | 25571688366
>> > > > decoder.invalid | RxPFRbond02 | 0
>> > > > decoder.ipv4 | RxPFRbond02 | 34483004
>> > > > decoder.ipv6 | RxPFRbond02 | 506
>> > > > decoder.ethernet | RxPFRbond02 | 33932611
>> > > > decoder.raw | RxPFRbond02 | 0
>> > > > decoder.null | RxPFRbond02 | 0
>> > > > decoder.sll | RxPFRbond02 | 0
>> > > > decoder.tcp | RxPFRbond02 | 24665968
>> > > > decoder.udp | RxPFRbond02 | 8600129
>> > > > decoder.sctp | RxPFRbond02 | 0
>> > > > decoder.icmpv4 | RxPFRbond02 | 113797
>> > > > decoder.icmpv6 | RxPFRbond02 | 0
>> > > > decoder.ppp | RxPFRbond02 | 0
>> > > > decoder.pppoe | RxPFRbond02 | 0
>> > > > decoder.gre | RxPFRbond02 | 0
>> > > > decoder.vlan | RxPFRbond02 | 0
>> > > > decoder.vlan_qinq | RxPFRbond02 | 0
>> > > > decoder.teredo | RxPFRbond02 | 506
>> > > > decoder.ipv4_in_ipv6 | RxPFRbond02 | 0
>> > > > decoder.ipv6_in_ipv6 | RxPFRbond02 | 0
>> > > > decoder.mpls | RxPFRbond02 | 0
>> > > > decoder.avg_pkt_size | RxPFRbond02 | 753
>> > > > decoder.max_pkt_size | RxPFRbond02 | 1514
>> > > > decoder.erspan | RxPFRbond02 | 0
>> > > > flow.memcap | RxPFRbond02 | 0
>> > > > defrag.ipv4.fragments | RxPFRbond02 | 1103110
>> > > > defrag.ipv4.reassembled | RxPFRbond02 | 550393
>> > > > defrag.ipv4.timeouts | RxPFRbond02 | 0
>> > > > defrag.ipv6.fragments | RxPFRbond02 | 0
>> > > > defrag.ipv6.reassembled | RxPFRbond02 | 0
>> > > > defrag.ipv6.timeouts | RxPFRbond02 | 0
>> > > > defrag.max_frag_hits | RxPFRbond02 | 0
>> > > > tcp.sessions | RxPFRbond02 | 172432
>> > > > tcp.ssn_memcap_drop | RxPFRbond02 | 0
>> > > > tcp.pseudo | RxPFRbond02 | 79224
>> > > > tcp.pseudo_failed | RxPFRbond02 | 0
>> > > > tcp.invalid_checksum | RxPFRbond02 | 0
>> > > > tcp.no_flow | RxPFRbond02 | 0
>> > > > tcp.syn | RxPFRbond02 | 183912
>> > > > tcp.synack | RxPFRbond02 | 150219
>> > > > tcp.rst | RxPFRbond02 | 143693
>> > > > tcp.segment_memcap_drop | RxPFRbond02 | 0
>> > > > tcp.stream_depth_reached | RxPFRbond02 | 105
>> > > > tcp.reassembly_gap | RxPFRbond02 | 4710
>> > > > detect.alert | RxPFRbond02 | 3469
>> > > > capture.kernel_packets | RxPFRbond03 | 38750498
>> > > > capture.kernel_drops | RxPFRbond03 | 1511800
>> > > > decoder.pkts | RxPFRbond03 | 38762341
>> > > > decoder.bytes | RxPFRbond03 | 32714534213
>> > > > decoder.invalid | RxPFRbond03 | 0
>> > > > decoder.ipv4 | RxPFRbond03 | 39299710
>> > > > decoder.ipv6 | RxPFRbond03 | 512
>> > > > decoder.ethernet | RxPFRbond03 | 38762341
>> > > > decoder.raw | RxPFRbond03 | 0
>> > > > decoder.null | RxPFRbond03 | 0
>> > > > decoder.sll | RxPFRbond03 | 0
>> > > > decoder.tcp | RxPFRbond03 | 21943466
>> > > > decoder.udp | RxPFRbond03 | 15992492
>> > > > decoder.sctp | RxPFRbond03 | 0
>> > > > decoder.icmpv4 | RxPFRbond03 | 178089
>> > > > decoder.icmpv6 | RxPFRbond03 | 0
>> > > > decoder.ppp | RxPFRbond03 | 0
>> > > > decoder.pppoe | RxPFRbond03 | 0
>> > > > decoder.gre | RxPFRbond03 | 0
>> > > > decoder.vlan | RxPFRbond03 | 0
>> > > > decoder.vlan_qinq | RxPFRbond03 | 0
>> > > > decoder.teredo | RxPFRbond03 | 512
>> > > > decoder.ipv4_in_ipv6 | RxPFRbond03 | 0
>> > > > decoder.ipv6_in_ipv6 | RxPFRbond03 | 0
>> > > > decoder.mpls | RxPFRbond03 | 0
>> > > > decoder.avg_pkt_size | RxPFRbond03 | 843
>> > > > decoder.max_pkt_size | RxPFRbond03 | 1514
>> > > > decoder.erspan | RxPFRbond03 | 0
>> > > > flow.memcap | RxPFRbond03 | 0
>> > > > defrag.ipv4.fragments | RxPFRbond03 | 1078454
>> > > > defrag.ipv4.reassembled | RxPFRbond03 | 537369
>> > > > defrag.ipv4.timeouts | RxPFRbond03 | 0
>> > > > defrag.ipv6.fragments | RxPFRbond03 | 0
>> > > > defrag.ipv6.reassembled | RxPFRbond03 | 0
>> > > > defrag.ipv6.timeouts | RxPFRbond03 | 0
>> > > > defrag.max_frag_hits | RxPFRbond03 | 0
>> > > > tcp.sessions | RxPFRbond03 | 169832
>> > > > tcp.ssn_memcap_drop | RxPFRbond03 | 0
>> > > > tcp.pseudo | RxPFRbond03 | 78504
>> > > > tcp.pseudo_failed | RxPFRbond03 | 0
>> > > > tcp.invalid_checksum | RxPFRbond03 | 0
>> > > > tcp.no_flow | RxPFRbond03 | 0
>> > > > tcp.syn | RxPFRbond03 | 181453
>> > > > tcp.synack | RxPFRbond03 | 147649
>> > > > tcp.rst | RxPFRbond03 | 139792
>> > > > tcp.segment_memcap_drop | RxPFRbond03 | 0
>> > > > tcp.stream_depth_reached | RxPFRbond03 | 94
>> > > > tcp.reassembly_gap | RxPFRbond03 | 2567
>> > > > detect.alert | RxPFRbond03 | 3416
>> > > > capture.kernel_packets | RxPFRbond04 | 63727760
>> > > > capture.kernel_drops | RxPFRbond04 | 3046651
>> > > > decoder.pkts | RxPFRbond04 | 63747722
>> > > > decoder.bytes | RxPFRbond04 | 55373084583
>> > > > decoder.invalid | RxPFRbond04 | 0
>> > > > decoder.ipv4 | RxPFRbond04 | 64056225
>> > > > decoder.ipv6 | RxPFRbond04 | 487
>> > > > decoder.ethernet | RxPFRbond04 | 63747722
>> > > > decoder.raw | RxPFRbond04 | 0
>> > > > decoder.null | RxPFRbond04 | 0
>> > > > decoder.sll | RxPFRbond04 | 0
>> > > > decoder.tcp | RxPFRbond04 | 55855784
>> > > > decoder.udp | RxPFRbond04 | 7447497
>> > > > decoder.sctp | RxPFRbond04 | 0
>> > > > decoder.icmpv4 | RxPFRbond04 | 133539
>> > > > decoder.icmpv6 | RxPFRbond04 | 0
>> > > > decoder.ppp | RxPFRbond04 | 0
>> > > > decoder.pppoe | RxPFRbond04 | 0
>> > > > decoder.gre | RxPFRbond04 | 0
>> > > > decoder.vlan | RxPFRbond04 | 0
>> > > > decoder.vlan_qinq | RxPFRbond04 | 0
>> > > > decoder.teredo | RxPFRbond04 | 487
>> > > > decoder.ipv4_in_ipv6 | RxPFRbond04 | 0
>> > > > decoder.ipv6_in_ipv6 | RxPFRbond04 | 0
>> > > > decoder.mpls | RxPFRbond04 | 0
>> > > > decoder.avg_pkt_size | RxPFRbond04 | 868
>> > > > decoder.max_pkt_size | RxPFRbond04 | 1514
>> > > > decoder.erspan | RxPFRbond04 | 0
>> > > > flow.memcap | RxPFRbond04 | 0
>> > > > defrag.ipv4.fragments | RxPFRbond04 | 619405
>> > > > defrag.ipv4.reassembled | RxPFRbond04 | 308503
>> > > > defrag.ipv4.timeouts | RxPFRbond04 | 0
>> > > > defrag.ipv6.fragments | RxPFRbond04 | 0
>> > > > defrag.ipv6.reassembled | RxPFRbond04 | 0
>> > > > defrag.ipv6.timeouts | RxPFRbond04 | 0
>> > > > defrag.max_frag_hits | RxPFRbond04 | 0
>> > > > tcp.sessions | RxPFRbond04 | 171368
>> > > > tcp.ssn_memcap_drop | RxPFRbond04 | 0
>> > > > tcp.pseudo | RxPFRbond04 | 78609
>> > > > tcp.pseudo_failed | RxPFRbond04 | 0
>> > > > tcp.invalid_checksum | RxPFRbond04 | 0
>> > > > tcp.no_flow | RxPFRbond04 | 0
>> > > > tcp.syn | RxPFRbond04 | 182409
>> > > > tcp.synack | RxPFRbond04 | 149124
>> > > > tcp.rst | RxPFRbond04 | 143473
>> > > > tcp.segment_memcap_drop | RxPFRbond04 | 0
>> > > > tcp.stream_depth_reached | RxPFRbond04 | 82
>> > > > tcp.reassembly_gap | RxPFRbond04 | 35459
>> > > > detect.alert | RxPFRbond04 | 3770
>> > > > flow_mgr.closed_pruned | FlowManagerThread | 310602
>> > > > flow_mgr.new_pruned | FlowManagerThread | 549722
>> > > > flow_mgr.est_pruned | FlowManagerThread | 380334
>> > > > flow.spare | FlowManagerThread | 799999
>> > > > flow.emerg_mode_entered | FlowManagerThread | 0
>> > > > flow.emerg_mode_over | FlowManagerThread | 0
>> > > > flow.tcp_reuse | FlowManagerThread | 237
>> > > > flow_mgr.closed_pruned | FlowManagerThread | 308878
>> > > > flow_mgr.new_pruned | FlowManagerThread | 544586
>> > > > flow_mgr.est_pruned | FlowManagerThread | 379393
>> > > > flow.spare | FlowManagerThread | 799402
>> > > > flow.emerg_mode_entered | FlowManagerThread | 0
>> > > > flow.emerg_mode_over | FlowManagerThread | 0
>> > > > flow.tcp_reuse | FlowManagerThread | 252
>> > > > tcp.memuse | Global | 439248976
>> > > > tcp.reassembly_memuse | Global | 1717630000
>> > > > dns.memuse | Global | 476478
>> > > > dns.memcap_state | Global | 0
>> > > > dns.memcap_global | Global | 0
>> > > > http.memuse | Global | 536216
>> > > > http.memcap | Global | 0
>> > > > flow.memuse | Global | 237040288
>> > > >
>> > > >
>> > > > _______________________________________________
>> > > > Suricata IDS Users mailing list:
>> > > > oisf-users at openinfosecfoundation.org
>> > > > Site: http://suricata-ids.org | Support:
>> > > > http://suricata-ids.org/support/
>> > > > List:
>> > > > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> > > > Suricata User Conference November 9-11 in Washington, DC:
>> > > > http://oisfevents.net
>> > >
>> > >
>> > >
>> > > --
>> > > Regards,
>> > > Peter Manev
>>
>>
>>
>>
>> --
>> Regards,
>> Peter Manev



-- 
Regards,
Peter Manev

More information about the Oisf-users mailing list