[Oisf-users] Tuning Suricata (2.0beta1) -- no rules and lots of packet loss

[rmkml]

Hi Tritium and Cooper,

Maybe it's related to recently speak on list for new dns preproc cause 
drop, I don't known if it possible to disable actually.
Maybe back to v1.4.4 for comparing ?

Regards
@Rmkml


On Wed, 14 Aug 2013, Cooper F. Nelson wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> #Edit:  I wrote the following before checking your suricata.yaml.  Try
> #not setting the "buffer-size:" directive under the af-packet
> #configuration.  Setting that seems to cause packet drops in AF_PACKET
> #+ mmap mode.
>
> What does your traffic look like?  Do you have any high volume, single
> flows?  e.g. two hosts doing a sustained >500mbit transfer?
>
> Suricata has an inherent limitation in that even on a tuned,
> high-performance system it can only process so many packets per second,
> per thread.  So if one (or more flows) are saturating a given core, the
> ring buffer will eventually fill and packets will be dropped.
>
> You can see this behavior using 'top' (press '1' to see all cores).
>
> This is my production box.  The cores that are 0% idle are at capacity
> and will eventually drop packets if the active flows fill the ring buffer:
>
>> top - 17:36:00 up 22 days, 18:12,  6 users,  load average: 14.71, 14.38, 14.34
>> Tasks: 211 total,   1 running, 210 sleeping,   0 stopped,   0 zombie
>> %Cpu0  : 72.8 us,  1.3 sy,  0.0 ni, 14.6 id,  0.0 wa,  0.0 hi, 11.3 si,  0.0 st
>> %Cpu1  : 89.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 10.3 si,  0.0 st
>> %Cpu2  : 89.4 us,  0.3 sy,  0.0 ni,  1.0 id,  0.0 wa,  0.0 hi,  9.3 si,  0.0 st
>> %Cpu3  : 88.4 us,  1.0 sy,  0.0 ni,  2.7 id,  0.0 wa,  0.0 hi,  8.0 si,  0.0 st
>> %Cpu4  : 80.4 us,  6.3 sy,  0.0 ni,  5.6 id,  0.0 wa,  0.0 hi,  7.6 si,  0.0 st
>> %Cpu5  : 80.7 us,  0.3 sy,  0.0 ni,  9.3 id,  0.0 wa,  0.0 hi,  9.6 si,  0.0 st
>> %Cpu6  : 90.0 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi, 10.0 si,  0.0 st
>> %Cpu7  : 91.4 us,  0.0 sy,  0.0 ni,  2.0 id,  0.0 wa,  0.0 hi,  6.6 si,  0.0 st
>> %Cpu8  : 79.1 us,  1.0 sy,  0.0 ni, 11.3 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
>> %Cpu9  : 67.8 us,  1.3 sy,  0.0 ni, 24.3 id,  0.0 wa,  0.0 hi,  6.6 si,  0.0 st
>> %Cpu10 : 90.0 us,  0.0 sy,  0.0 ni,  0.3 id,  0.0 wa,  0.0 hi,  9.6 si,  0.0 st
>> %Cpu11 : 90.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  9.0 si,  0.0 st
>> %Cpu12 : 85.4 us,  0.7 sy,  0.0 ni,  5.3 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
>> %Cpu13 : 68.1 us,  0.3 sy,  0.0 ni, 23.3 id,  0.0 wa,  0.0 hi,  8.3 si,  0.0 st
>> %Cpu14 : 91.4 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  8.6 si,  0.0 st
>> %Cpu15 : 69.8 us,  2.0 sy,  0.0 ni, 20.9 id,  0.0 wa,  0.0 hi,  7.3 si,  0.0 st
>> KiB Mem:  49457008 total, 49298704 used,   158304 free,   188764 buffers
>> KiB Swap:        0 total,        0 used,        0 free, 24121808 cached
>>
>>   PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
>>  3991 root      20   0 22.606g 0.021t 0.013t S  1400 45.50   3352:06 /usr/bin/suric+
>
> There are some things you can do addresses this.
>
> One, use AF_PACKET + mmap mode and set a large ring buffer:
>
>>     ring-size: 524288
>
> I've found if you go too far above this suricata will generate errors
> about allocating memory (which it will eventually recover from).  I
> believe this is an issue with the linux kernel and not suricata.  Be
> careful not to run your box out of memory.
>
> As mentioned, you should also set large receive queues via sysctl.conf:
>
>> net.core.netdev_max_backlog = 10000000
>> net.core.rmem_default = 1073741824
>> net.core.rmem_max = 1073741824
>
> You should also set short timeouts (e.g. 5 seconds) and lower the stream
> depth.  I have mine set to 2mb.
>
> Another thing you can do is if you can identify the large, boring data
> flows (like backups or other bulk transfers) you can use BPF filters to
> prevent those packets from being processed.
>
> - -Coop
>
> On 8/14/2013 10:26 AM, Tritium Cat wrote:
>>
>> Where I'm confused most is why Suricata is dropping so many packets with no
>> rules enabled.  The "AF-Packet" link below used this approach to find a
>> stable point before adding rules.  I've tuned the Intel cards as
>> recommended with setpci, ethtool, and ixgbe parameters.  The system has
>> also been tuned with various sysctl tweaks to match other's recommendations
>> (_rmem, _wmem, backlog, etc...) as well as set_irq_affinity to balance the
>> interrupts among all CPU cores.  (see attached files)
>>
>> Any help is much appreciated... thanks !
>>
>> --TC
>>
>
>
> - --
> Cooper Nelson
> Network Security Analyst
> UCSD ACT Security Team
> cnelson at ucsd.edu x41042