[Oisf-users] Packet Loss
Yasha Zislin
coolyasha at hotmail.com
Fri Jun 6 16:56:36 UTC 2014
So running with 8 threads per each feed produced the same outcome. After certain period of time Free Slots on each thread went to 0 and packet drop started to occur.
Here is a sample from stats.log
capture.kernel_packets | RxPFReth158 | 76566261
capture.kernel_drops | RxPFReth158 | 4420357
dns.memuse | RxPFReth158 | 20638
dns.memcap_state | RxPFReth158 | 0
dns.memcap_global | RxPFReth158 | 0
decoder.pkts | RxPFReth158 | 76566265
decoder.bytes | RxPFReth158 | 53493573115
decoder.invalid | RxPFReth158 | 6396180
decoder.ipv4 | RxPFReth158 | 76466893
decoder.ipv6 | RxPFReth158 | 31305
decoder.ethernet | RxPFReth158 | 76566265
decoder.raw | RxPFReth158 | 0
decoder.sll | RxPFReth158 | 0
decoder.tcp | RxPFReth158 | 67103643
decoder.udp | RxPFReth158 | 2909731
decoder.sctp | RxPFReth158 | 0
decoder.icmpv4 | RxPFReth158 | 74218
decoder.icmpv6 | RxPFReth158 | 0
decoder.ppp | RxPFReth158 | 0
decoder.pppoe | RxPFReth158 | 0
decoder.gre | RxPFReth158 | 0
decoder.vlan | RxPFReth158 | 0
decoder.vlan_qinq | RxPFReth158 | 0
decoder.teredo | RxPFReth158 | 320
decoder.ipv4_in_ipv6 | RxPFReth158 | 0
decoder.ipv6_in_ipv6 | RxPFReth158 | 0
decoder.avg_pkt_size | RxPFReth158 | 698
decoder.max_pkt_size | RxPFReth158 | 1514
defrag.ipv4.fragments | RxPFReth158 | 47620
defrag.ipv4.reassembled | RxPFReth158 | 57
defrag.ipv4.timeouts | RxPFReth158 | 0
defrag.ipv6.fragments | RxPFReth158 | 0
defrag.ipv6.reassembled | RxPFReth158 | 0
defrag.ipv6.timeouts | RxPFReth158 | 0
defrag.max_frag_hits | RxPFReth158 | 0
tcp.sessions | RxPFReth158 | 506051
tcp.ssn_memcap_drop | RxPFReth158 | 0
tcp.pseudo | RxPFReth158 | 43112
tcp.invalid_checksum | RxPFReth158 | 0
tcp.no_flow | RxPFReth158 | 0
tcp.reused_ssn | RxPFReth158 | 310
tcp.memuse | RxPFReth158 | 101392
tcp.syn | RxPFReth158 | 559759
tcp.synack | RxPFReth158 | 509133
tcp.rst | RxPFReth158 | 9413638
tcp.segment_memcap_drop | RxPFReth158 | 0
tcp.stream_depth_reached | RxPFReth158 | 86
tcp.reassembly_memuse | RxPFReth158 | 16419280000
tcp.reassembly_gap | RxPFReth158 | 138076
http.memuse | RxPFReth158 | 164323
http.memcap | RxPFReth158 | 0
detect.alert | RxPFReth158 | 646
flow_mgr.closed_pruned | FlowManagerThread | 21417560
flow_mgr.new_pruned | FlowManagerThread | 6460672
flow_mgr.est_pruned | FlowManagerThread | 3364287
flow.memuse | FlowManagerThread | 11722958144
flow.spare | FlowManagerThread | 40001633
flow.emerg_mode_entered | FlowManagerThread | 0
flow.emerg_mode_over | FlowManagerThread | 0
Packet loss is 5% at this moment but since Free Slots are at 0, current packet drop is higher than that.
I've noticed fragmentation in this log but i believe it is insignificant.
From: coolyasha at hotmail.com
To: petermanev at gmail.com; cnelson at ucsd.edu
Date: Fri, 6 Jun 2014 15:13:18 +0000
CC: oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Packet Loss
Cooper/Peter/Victor,
Thank you for detailed response. I will comment on everything in one response.
I have reconfigured Suricata to run 8 threads for each interface instead of 16. Will see how it goes. I've tried this before and noticed that packet loss occurs faster but I've changed settings many times, so I will test again. Just to point out, with 32 threads none of my CPUs reach 100%. And I get more buffers with PF_RING (ie Slots).
In regards to increasing buffers like for Stream. I kept running out of ideas where my bottleneck is so kept increasing since I have plenty of RAM. I assume this wont hurt anything but the RAM allocation.
I've also done profilling on my traffic. It is 99.9% HTTP.
The stats that I have provided do show good packet loss (ie 0.22%) and I would be ok with this if it was the whole time.
So whenever free slots come to 0 on all threads, packet loss starts to go above 20%. Then after some time, free slots go back to full amount and packet drop stops. Over time stats improve. I am trying to figure out why I get this sudden drop in packets. What I dont understand is why my Free Slots get saturated so much.
I've configured min num slots on PF_RING to be 400,000. I know it is way higher than 65k value that it shows in every guide. But this worked. It just started to use a gig of ram for each thread. Looking at stats for Free Num Slots in each thread, it is set to 680k. It seems to work except for saturation described above. I've tried using 65k number instead but saturation occurs even faster.
I've also ran PF_COUNT tool to see packet loss and it looks good. No packets lost.
SPAN Port is 10 gig fiber card and the actual traffic does not go above 1gig. Linux kernel does not report any packet loss on these interfaces.
I've also disabled checksum offloading on the nic and in suricata config checksum checks are disabled as well.
> Date: Thu, 5 Jun 2014 20:49:53 +0200
> Subject: Re: [Oisf-users] Packet Loss
> From: petermanev at gmail.com
> To: cnelson at ucsd.edu
> CC: coolyasha at hotmail.com; oisf-users at lists.openinfosecfoundation.org
>
> On Wed, Jun 4, 2014 at 6:52 PM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Not sure if its an option, but you might want to consider doubling your
> > cores. The published guides tend to use a 16 core system, but the
> > reality is that if you are really going for a multi-gigabit deployment
> > and zero packet drops you are going to need 32 cores at least.
> >
> > On 6/4/2014 8:04 AM, Yasha Zislin wrote:
> >> To summarize my environment and set up.
> >> I have one server (16 CPU Cores, 132gb of ram) with two span ports which
> >> I monitor with PF_RING and Suricata 2.0.1.
> >> I've configured all of the buffers pretty high. When my suricata is
> >> running it is using almost 70 gigs of RAM.
> >
> >
> > - --
> > Cooper Nelson
> > Network Security Analyst
> > UCSD ACT Security Team
> > cnelson at ucsd.edu x41042
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.17 (MingW32)
> > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >
> > iQEcBAEBAgAGBQJTj07DAAoJEKIFRYQsa8FWDNAH/1dGJ6iXhGtDGxCSDdSwT/NP
> > PrrF0AqJ8u97KJUZAIbVZOW5YZaFHGfW/HJuq8bpTokWEAYva7dfwIHG8MyfWT5A
> > 9ED9aZvdU+aQ0vWEkPE6pAmNj8cbLHSsUPOM1vPYlCFbJmIP9DJGA6SwwnuWNn8h
> > ChAZR5lCxdRkIEhfNFg5/fiSwHuFPZCHTS8+Qt3IyLuToQbApG0UqBM5zkOrqJm7
> > DohucMg5L28hU+xDB3m2Jv1kR7wQ5icmCME+fxwKiPWZd+rVa/EDnemnfQpiigGx
> > eOBc2Tsw+sncfv5KIcKrC9zKu5RVrf3r4mJwt5hphG66y3RbWkKazbzT+jm9Pu8=
> > =FkAe
> > -----END PGP SIGNATURE-----
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
>
>
> You have 16 cores but you have provided 32 threads in total (16 eth15
> and 16 eth17)
>
> I would suggest try using 8 and 8 respectively - if you get the same
> amount of traffic on both ethX
>
> "I think it should be able to handle the load with 0 packet loss" -
> the fact that you have a lot of HW does not mean you should get 0
> packet loss.
>
> The loss may occur at any given point in the traffic and/or on the way
> to the mirror port - and if it is a stream, depending where(and how
> often) in the stream the packet lost occurs, the whole stream might be
> dropped/flushed by the engine.
>
> I mean packet loss can happen for a reason and just one of that reason
> might be memcap buffers full - but there could be packet loss related
> to a whole lot of other stuff that has nothing to do with memcap
> buffers and/or processing...
>
> Judging from your output of stats.log -
>
> capture.kernel_packets | RxPFReth1516 | 86380604
> capture.kernel_drops | RxPFReth1516 | 195847
>
> this is actually - 0.22% loss..... for that particular thread. If
> all of them share the same result (with 22k rules) ... I would say you
> are doing quite alright.
>
> Another observation - you have 64 million sessions preallocated
> "3/6/2014 -- 15:40:31 - <Info> - stream "prealloc-sessions": 2000000
> (per thread)"
> eth15 - 16x2mil, eth17 - 16x2mil
>
> Just a suggestion (word of advise) if I may:
> You mention you have 180k packets per second at peak times - imagine
> for second that every single packet initiates a session and you would
> see that it is a bit of an excessive setting to have 64 mill prealloc
> sessions.
>
> I would suggest that profiling and exploring(knowing) your traffic is
> a much better approach than just adding 0s at the memcaps in the
> configuration settings.
>
>
>
> --
> Regards,
> Peter Manev
_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
OISF: http://www.openinfosecfoundation.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140606/798d3543/attachment-0002.html>
More information about the Oisf-users
mailing list