[Oisf-users] Tuning Suricata (2.0beta1) -- no rules and lots of packet loss

Wed Aug 14 18:10:02 UTC 2013

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

#Edit:  I wrote the following before checking your suricata.yaml.  Try
#not setting the "buffer-size:" directive under the af-packet
#configuration.  Setting that seems to cause packet drops in AF_PACKET
#+ mmap mode.

What does your traffic look like?  Do you have any high volume, single
flows?  e.g. two hosts doing a sustained >500mbit transfer?

Suricata has an inherent limitation in that even on a tuned,
high-performance system it can only process so many packets per second,
per thread.  So if one (or more flows) are saturating a given core, the
ring buffer will eventually fill and packets will be dropped.

You can see this behavior using 'top' (press '1' to see all cores).

This is my production box.  The cores that are 0% idle are at capacity
and will eventually drop packets if the active flows fill the ring buffer:

> top - 17:36:00 up 22 days, 18:12,  6 users,  load average: 14.71, 14.38, 14.34
> Tasks: 211 total,   1 running, 210 sleeping,   0 stopped,   0 zombie
> %Cpu0  : 72.8 us,  1.3 sy,  0.0 ni, 14.6 id,  0.0 wa,  0.0 hi, 11.3 si,  0.0 st
> %Cpu1  : 89.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 10.3 si,  0.0 st
> %Cpu2  : 89.4 us,  0.3 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  9.3 si,  0.0 st
> %Cpu3  : 88.4 us,  1.0 sy,  0.0 ni,  2.7 id,  0.0 wa,  0.0 hi,  8.0 si,  0.0 st
> %Cpu4  : 80.4 us,  6.3 sy,  0.0 ni,  5.6 id,  0.0 wa,  0.0 hi,  7.6 si,  0.0 st
> %Cpu5  : 80.7 us,  0.3 sy,  0.0 ni,  9.3 id,  0.0 wa,  0.0 hi,  9.6 si,  0.0 st
> %Cpu6  : 90.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 10.0 si,  0.0 st
> %Cpu7  : 91.4 us,  0.0 sy,  0.0 ni,  2.0 id,  0.0 wa,  0.0 hi,  6.6 si,  0.0 st
> %Cpu8  : 79.1 us,  1.0 sy,  0.0 ni, 11.3 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
> %Cpu9  : 67.8 us,  1.3 sy,  0.0 ni, 24.3 id,  0.0 wa,  0.0 hi,  6.6 si,  0.0 st
> %Cpu10 : 90.0 us,  0.0 sy,  0.0 ni,  0.3 id,  0.0 wa,  0.0 hi,  9.6 si,  0.0 st
> %Cpu11 : 90.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  9.0 si,  0.0 st
> %Cpu12 : 85.4 us,  0.7 sy,  0.0 ni,  5.3 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
> %Cpu13 : 68.1 us,  0.3 sy,  0.0 ni, 23.3 id,  0.0 wa,  0.0 hi,  8.3 si,  0.0 st
> %Cpu14 : 91.4 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
> %Cpu15 : 69.8 us,  2.0 sy,  0.0 ni, 20.9 id,  0.0 wa,  0.0 hi,  7.3 si,  0.0 st
> KiB Mem:  49457008 total, 49298704 used,   158304 free,   188764 buffers
> KiB Swap:        0 total,        0 used,        0 free, 24121808 cached
> 
>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
>  3991 root      20   0 22.606g 0.021t 0.013t S  1400 45.50   3352:06 /usr/bin/suric+

There are some things you can do addresses this.

One, use AF_PACKET + mmap mode and set a large ring buffer:

>     ring-size: 524288

I've found if you go too far above this suricata will generate errors
about allocating memory (which it will eventually recover from).  I
believe this is an issue with the linux kernel and not suricata.  Be
careful not to run your box out of memory.

As mentioned, you should also set large receive queues via sysctl.conf:

> net.core.netdev_max_backlog = 10000000
> net.core.rmem_default = 1073741824
> net.core.rmem_max = 1073741824

You should also set short timeouts (e.g. 5 seconds) and lower the stream
depth.  I have mine set to 2mb.

Another thing you can do is if you can identify the large, boring data
flows (like backups or other bulk transfers) you can use BPF filters to
prevent those packets from being processed.

- -Coop

On 8/14/2013 10:26 AM, Tritium Cat wrote:
> 
> Where I'm confused most is why Suricata is dropping so many packets with no
> rules enabled.  The "AF-Packet" link below used this approach to find a
> stable point before adding rules.  I've tuned the Intel cards as
> recommended with setpci, ethtool, and ixgbe parameters.  The system has
> also been tuned with various sysctl tweaks to match other's recommendations
> (_rmem, _wmem, backlog, etc...) as well as set_irq_affinity to balance the
> interrupts among all CPU cores.  (see attached files)
> 
> Any help is much appreciated... thanks !
> 
> --TC
> 

- -- 
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSC8f6AAoJEKIFRYQsa8FW7awIAMRRMWGBCrgOm9mj6NwE1rF8
QL2bssAxi07twjLEpmkRGWVevm8i8XbkgZxBNUJJXlAqoyBGPKixurZwLYnNvJ89
BOFMvynHF5KhTqtioUFS8J98IcB1+ZuetLHxgCAfVhBAPVgJJ6XlOqPdMEjrOExJ
3TYL0oeN2FxQ5w9ufpx8vD71c88YCLmlTnIOYyCWHUu3yxqJ/vOaF576wiXBgxLh
63cwvje2j/8fOBQvPAUQahBHa3X288XKS4gC5DvuH3SCCW94KpVYGhZYuG4U/dT5
98yh/QLzRA0xA0MyrYx0vMfF7ZylhiWmYxjIBKw11/mGXQAq+GN+Jx+8zhi4NIM=
=+VwZ
-----END PGP SIGNATURE-----