<div dir="ltr">Hello.<div><br></div><div>Yes good idea, I just read about that problem here:<div><br></div><div><a href="https://lists.openinfosecfoundation.org/pipermail/oisf-users/2013-July/002724.html">https://lists.openinfosecfoundation.org/pipermail/oisf-users/2013-July/002724.html</a><br>
</div><div><br></div><div>I'll give 1.4.4 a try.</div></div><div><br></div><div>--TC</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Aug 14, 2013 at 11:43 AM, rmkml <span dir="ltr"><<a href="mailto:rmkml@yahoo.fr" target="_blank">rmkml@yahoo.fr</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Tritium and Cooper,<br>
<br>
Maybe it's related to recently speak on list for new dns preproc cause drop, I don't known if it possible to disable actually.<br>
Maybe back to v1.4.4 for comparing ?<br>
<br>
Regards<span class="HOEnZb"><font color="#888888"><br>
@Rmkml</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
<br>
On Wed, 14 Aug 2013, Cooper F. Nelson wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
#Edit: I wrote the following before checking your suricata.yaml. Try<br>
#not setting the "buffer-size:" directive under the af-packet<br>
#configuration. Setting that seems to cause packet drops in AF_PACKET<br>
#+ mmap mode.<br>
<br>
What does your traffic look like? Do you have any high volume, single<br>
flows? e.g. two hosts doing a sustained >500mbit transfer?<br>
<br>
Suricata has an inherent limitation in that even on a tuned,<br>
high-performance system it can only process so many packets per second,<br>
per thread. So if one (or more flows) are saturating a given core, the<br>
ring buffer will eventually fill and packets will be dropped.<br>
<br>
You can see this behavior using 'top' (press '1' to see all cores).<br>
<br>
This is my production box. The cores that are 0% idle are at capacity<br>
and will eventually drop packets if the active flows fill the ring buffer:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
top - 17:36:00 up 22 days, 18:12, 6 users, load average: 14.71, 14.38, 14.34<br>
Tasks: 211 total, 1 running, 210 sleeping, 0 stopped, 0 zombie<br>
%Cpu0 : 72.8 us, 1.3 sy, 0.0 ni, 14.6 id, 0.0 wa, 0.0 hi, 11.3 si, 0.0 st<br>
%Cpu1 : 89.7 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 10.3 si, 0.0 st<br>
%Cpu2 : 89.4 us, 0.3 sy, 0.0 ni, 1.0 id, 0.0 wa, 0.0 hi, 9.3 si, 0.0 st<br>
%Cpu3 : 88.4 us, 1.0 sy, 0.0 ni, 2.7 id, 0.0 wa, 0.0 hi, 8.0 si, 0.0 st<br>
%Cpu4 : 80.4 us, 6.3 sy, 0.0 ni, 5.6 id, 0.0 wa, 0.0 hi, 7.6 si, 0.0 st<br>
%Cpu5 : 80.7 us, 0.3 sy, 0.0 ni, 9.3 id, 0.0 wa, 0.0 hi, 9.6 si, 0.0 st<br>
%Cpu6 : 90.0 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 10.0 si, 0.0 st<br>
%Cpu7 : 91.4 us, 0.0 sy, 0.0 ni, 2.0 id, 0.0 wa, 0.0 hi, 6.6 si, 0.0 st<br>
%Cpu8 : 79.1 us, 1.0 sy, 0.0 ni, 11.3 id, 0.0 wa, 0.0 hi, 8.6 si, 0.0 st<br>
%Cpu9 : 67.8 us, 1.3 sy, 0.0 ni, 24.3 id, 0.0 wa, 0.0 hi, 6.6 si, 0.0 st<br>
%Cpu10 : 90.0 us, 0.0 sy, 0.0 ni, 0.3 id, 0.0 wa, 0.0 hi, 9.6 si, 0.0 st<br>
%Cpu11 : 90.7 us, 0.3 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 9.0 si, 0.0 st<br>
%Cpu12 : 85.4 us, 0.7 sy, 0.0 ni, 5.3 id, 0.0 wa, 0.0 hi, 8.6 si, 0.0 st<br>
%Cpu13 : 68.1 us, 0.3 sy, 0.0 ni, 23.3 id, 0.0 wa, 0.0 hi, 8.3 si, 0.0 st<br>
%Cpu14 : 91.4 us, 0.0 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 8.6 si, 0.0 st<br>
%Cpu15 : 69.8 us, 2.0 sy, 0.0 ni, 20.9 id, 0.0 wa, 0.0 hi, 7.3 si, 0.0 st<br>
KiB Mem: 49457008 total, 49298704 used, 158304 free, 188764 buffers<br>
KiB Swap: 0 total, 0 used, 0 free, 24121808 cached<br>
<br>
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<br>
3991 root 20 0 22.606g 0.021t 0.013t S 1400 45.50 3352:06 /usr/bin/suric+<br>
</blockquote>
<br>
There are some things you can do addresses this.<br>
<br>
One, use AF_PACKET + mmap mode and set a large ring buffer:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
ring-size: 524288<br>
</blockquote>
<br>
I've found if you go too far above this suricata will generate errors<br>
about allocating memory (which it will eventually recover from). I<br>
believe this is an issue with the linux kernel and not suricata. Be<br>
careful not to run your box out of memory.<br>
<br>
As mentioned, you should also set large receive queues via sysctl.conf:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
net.core.netdev_max_backlog = 10000000<br>
net.core.rmem_default = 1073741824<br>
net.core.rmem_max = 1073741824<br>
</blockquote>
<br>
You should also set short timeouts (e.g. 5 seconds) and lower the stream<br>
depth. I have mine set to 2mb.<br>
<br>
Another thing you can do is if you can identify the large, boring data<br>
flows (like backups or other bulk transfers) you can use BPF filters to<br>
prevent those packets from being processed.<br>
<br>
- -Coop<br>
<br>
On 8/14/2013 10:26 AM, Tritium Cat wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Where I'm confused most is why Suricata is dropping so many packets with no<br>
rules enabled. The "AF-Packet" link below used this approach to find a<br>
stable point before adding rules. I've tuned the Intel cards as<br>
recommended with setpci, ethtool, and ixgbe parameters. The system has<br>
also been tuned with various sysctl tweaks to match other's recommendations<br>
(_rmem, _wmem, backlog, etc...) as well as set_irq_affinity to balance the<br>
interrupts among all CPU cores. (see attached files)<br>
<br>
Any help is much appreciated... thanks !<br>
<br>
--TC<br>
<br>
</blockquote>
<br>
<br>
- --<br>
Cooper Nelson<br>
Network Security Analyst<br>
UCSD ACT Security Team<br>
<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a> x41042<br>
</blockquote>
</div></div></blockquote></div><br></div>