[Oisf-users] Packet Loss

[Cooper F. Nelson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

FYI, based on my experience with real-world traffic, there are scenarios
where a single high volume flow can overwhelm a single buffer/thread and
packets will be dropped.  I believe this occurs even if the stream
tracking depth is reached for that single flow, as the kernel is still
pushing packets to the ring buffer for suricata to process.

Another way to think about it is that there is a hard limit of the
number of packets-per-second each thread can process, per core.  The
only way to alleviate this currently is to use faster cores/bigger
buffers.  But I think the reality is that two hosts with a low-latency
gigabit network connection can saturate a single core/buffer easily.

One way to alleviate this is to identify this sort of traffic, (e.g.
NetFlix) and filter it out via a bpf filter.  As far as I can tell this
filter is applied by the kernel prior to sending packets to the buffer
for processing.  We use some pretty extreme bpf filters, which keeps our
kernel packet drops under 1%.

So, I think one way the suricata devs could fix this would be to move
the stream tracking from user space to kernel space, so packets past the
stream depth limit aren't pushed to the ring buffer for further
processing.  Of course, they may already be doing something like this or
it may currently be impossible given software limitations.

If you don't care about HTTP traffic past the server header you can
sample http flows via this bpf filter.  This will probably fix your
issue with packet drops, but will also break file extraction and most of
the http client sigs:

> (not port 80 or (tcp dst port 80 or (tcp src port 80 and (tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))

- -Coop

On 6/4/2014 8:04 AM, Yasha Zislin wrote:
> So my opinion is that there is some sort of buffer which gets
> overwhelmed by flows or sessions and Suricata just cant keep all of
> these sessions in memory.
> Most of my traffic is HTTP.
> At peak times, one SPAN port gets about 180k packets per second and the
> other is 150k packets per second. Bandwidth never goes over 1 gig.
> I am using PF_RING in Mode 0.



- -- 
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTjz66AAoJEKIFRYQsa8FW/WUH/RArAozFJtuMDUNugwl7tygp
JGwXabUv6+X2i5sqly8Ztl9QfAFr8rdZaH9L/JQD+qvmNrhPsRsKLKEbmEc+MU1G
BM4wum+YS6xzyqMyYfKjWRbdTlpCc30towZQwPEttExIqLuzqYX4xNJjlVpCnwLb
bR+U5P6dRT3wbyOEBP2oSHIi1klRQ6IoK5OjBz+S/hWpw9PkV+ngo62ZYRG2zkYC
7lOEGQ2nEabD4Mp3emN5zEzjnDBHAtBmij0oUUUkkFkPgqq3ZqRBOPVtxdladcng
KDm+42gwIuuaejf0z6bUrTkoDgUw64OsRuQLpuwV31QPX2spNvetmv+iACRL0cY=
=vOaa
-----END PGP SIGNATURE-----