<div dir="ltr"><div>We set "cluster-type: cluster_cpu" as suggested and CPU load lowered from 30% (average) to 5%!! But, the unbalance is still there. Also the UDP traffic is balanced now (sudo ethtool -N eth7 rx-flow-hash udp4 sdfn).<br>
</div><div><div><br> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br> 2299 root 22 2 55.8g 53g 51g R 80.1 28.6 5:12.04 AFPacketeth51 <br>
2331 root 20 0 55.8g 53g 51g R 19.9 28.6 1:19.97 FlowManagerThre <br> 2324 root 18 -2 55.8g 53g 51g S 16.4 28.6 1:13.06 AFPacketeth710 <br>
2315 root 18 -2 55.8g 53g 51g S 11.9 28.6 0:49.75 AFPacketeth71 <br> 2328 root 18 -2 55.8g 53g 51g S 11.9 28.6 0:55.22 AFPacketeth714 <br>
2316 root 18 -2 55.8g 53g 51g S 10.9 28.6 0:54.53 AFPacketeth72 <br> 2326 root 18 -2 55.8g 53g 51g S 10.9 28.6 0:45.33 AFPacketeth712 <br>
2317 root 18 -2 55.8g 53g 51g S 10.4 28.6 0:38.21 AFPacketeth73 <br> 2323 root 18 -2 55.8g 53g 51g S 9.9 28.6 0:44.72 AFPacketeth79 <br>
<br></div><div>Dropped kernel packets:<br></div><div><br>capture.kernel_drops | AFPacketeth51 | 449774742<br>capture.kernel_drops | AFPacketeth52 | 48573<br>capture.kernel_drops | AFPacketeth53 | 104763<br>
capture.kernel_drops | AFPacketeth54 | 108080<br>capture.kernel_drops | AFPacketeth55 | 95763<br>capture.kernel_drops | AFPacketeth56 | 105133<br>capture.kernel_drops | AFPacketeth57 | 103984<br>
capture.kernel_drops | AFPacketeth58 | 100208<br>capture.kernel_drops | AFPacketeth59 | 86704<br>capture.kernel_drops | AFPacketeth510 | 95995<br>capture.kernel_drops | AFPacketeth511 | 89633<br>
capture.kernel_drops | AFPacketeth512 | 94029<br>capture.kernel_drops | AFPacketeth513 | 95192<br>capture.kernel_drops | AFPacketeth514 | 106460<br>capture.kernel_drops | AFPacketeth515 | 109770<br>
capture.kernel_drops | AFPacketeth516 | 108373<br><br><br>idsuser@suricata:/var/log/suricata$ cat /etc/rc.local <br>#!/bin/sh -e<br>#<br># rc.local<br>#<br># This script is executed at the end of each multiuser runlevel.<br>
# Make sure that the script will "exit 0" on success or any other<br># value on error.<br>#<br># In order to enable or disable this script just change the execution<br># bits.<br>#<br># By default this script does nothing.<br>
<br>sudo sysctl -w net.core.rmem_max=536870912<br>sudo sysctl -w net.core.wmem_max=67108864<br>sudo sysctl -w net.ipv4.tcp_window_scaling=1<br>sudo sysctl -w net.core.netdev_max_backlog=1000000<br><br># Seteo tamaño de MMRBC en bus en 4K<br>
sudo setpci -d 8086:10fb e6.b=2e<br><br># sudo sysctl -w net.ipv4.tcp_rmem="4096 87380 67108864"<br># sudo sysctl -w net.ipv4.tcp_wmem="4096 87380 67108864"<br><br>sleep 2<br>sudo rmmod ixgbe<br>sleep 2<br>
sudo insmod /lib/modules/3.2.0-45-generic/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko FdirPballoc=3,3,3,3 RSS=16,16,16,16 DCA=2,2,2,2<br>sleep 2<br><br># Seteo ring size<br># sudo ethtool -G eth4 rx 4096<br>sudo ethtool -G eth5 rx 4096<br>
# sudo ethtool -G eth6 rx 4096<br>sudo ethtool -G eth7 rx 4096<br><br># Balanceo de carga de flows UDP<br># sudo ethtool -N eth4 rx-flow-hash udp4 sdfn<br>sudo ethtool -N eth5 rx-flow-hash udp4 sdfn<br># sudo ethtool -N eth6 rx-flow-hash udp4 sdfn<br>
sudo ethtool -N eth7 rx-flow-hash udp4 sdfn<br><br>sleep 2<br>sudo ksh /home/idsuser/ixgbe-3.14.5/scripts/set_irq_affinity eth4 eth5 eth6 eth7<br>sleep 2<br># sudo ifconfig eth4 up && sleep 1<br>sudo ifconfig eth5 up && sleep 1<br>
# sudo ifconfig eth6 up && sleep 1<br>sudo ifconfig eth7 up && sleep 1<br>sleep 5<br>sudo suricata -D -c /etc/suricata/suricata.yaml --af-packet<br>sleep 10<br>sudo barnyard2 -c /etc/suricata/barnyard2.conf -d /var/log/suricata -f unified2.alert -w /var/log/suricata/suricata.waldo -D<br>
exit 0<br><br></div></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/3 Fernando Sclavo <span dir="ltr"><<a href="mailto:fsclavo@gmail.com" target="_blank">fsclavo@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Correction to previous email: runmode IS set to workers<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra">
<br><br><div class="gmail_quote">2013/6/3 Fernando Sclavo <span dir="ltr"><<a href="mailto:fsclavo@gmail.com" target="_blank">fsclavo@gmail.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hi Peter/Eric, I will try "flow per cpu" and mail the results. Same to "workers", but if I don't mistake we has tried but CPU usage was very high.<br>
<br><br></div><div>Queues and IRQ affinity: each NIC has 16 queues, with IRQ assigned to one core each one (Intel driver script), and Suricata has CPU affinity enabled, confirmed that each thread keeps in their own core.<br>
<br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">2013/6/3 Eric Leblond <span dir="ltr"><<a href="mailto:eric@regit.org" target="_blank">eric@regit.org</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
Le lundi 03 juin 2013 à 15:54 +0200, Peter Manev a écrit :<br>
<div><div>><br>
><br>
><br>
> On Mon, Jun 3, 2013 at 3:34 PM, Fernando Sclavo <<a href="mailto:fsclavo@gmail.com" target="_blank">fsclavo@gmail.com</a>><br>
> wrote:<br>
> Hi all!<br>
> We are running Suricata 1.4.2 with two Intel x520 cards,<br>
> connected each one to the core switches on our datacenter<br>
> network. The average traffic is about 1~2Gbps per port.<br>
> As you can see on the following top output, there are some<br>
> threads significantly more loaded than others (AFPacketeth54<br>
> for example): these threads are continuously dropping kernel<br>
> packets. We raised kernel parameters (buffers and rmem, etc)<br>
> and lowered suricata timeouts flows to just a few seconds, but<br>
> we can't keep drops counter static when CPU goes to 99.9% for<br>
> a specific thread.<br>
> How can we do to balance the load better on all threads to<br>
> prevent this issue?<br>
><br>
> The server is a Dell R715 2x16 core AMD Opteron(tm) Processor<br>
> 6284, 192Gb RAM.<br>
><br>
> idsuser@suricata:~$ top -d2<br>
><br>
> top - 10:24:05 up 1 min, 2 users, load average: 4.49, 1.14,<br>
> 0.38<br>
> Tasks: 287 total, 15 running, 272 sleeping, 0 stopped, 0<br>
> zombie<br>
> Cpu(s): 30.3%us, 1.3%sy, 0.0%ni, 65.3%id, 0.0%wa, 0.0%hi,<br>
> 3.1%si, 0.0%st<br>
> Mem: 198002932k total, 59619020k used, 138383912k free,<br>
> 25644k buffers<br>
> Swap: 15624188k total, 0k used, 15624188k free,<br>
> 161068k cached<br>
><br>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+<br>
> COMMAND<br>
> 2309 root 18 -2 55.8g 54g 51g R 99.9 28.6 0:20.96<br>
> AFPacketeth54<br>
> 2314 root 18 -2 55.8g 54g 51g R 99.9 28.6 0:18.29<br>
> AFPacketeth59<br>
> 2318 root 18 -2 55.8g 54g 51g R 99.9 28.6 0:12.90<br>
> AFPacketeth513<br>
> 2319 root 18 -2 55.8g 54g 51g R 77.6 28.6 0:12.78<br>
> AFPacketeth514<br>
> 2307 root 20 0 55.8g 54g 51g S 66.6 28.6 0:21.25<br>
> AFPacketeth52<br>
> 2338 root 20 0 55.8g 54g 51g R 58.2 28.6 0:09.94<br>
> FlowManagerThre<br>
> 2310 root 18 -2 55.8g 54g 51g S 51.2 28.6 0:15.35<br>
> AFPacketeth55<br>
> 2320 root 18 -2 55.8g 54g 51g R 50.2 28.6 0:07.83<br>
> AFPacketeth515<br>
> 2313 root 18 -2 55.8g 54g 51g S 48.7 28.6 0:11.66<br>
> AFPacketeth58<br>
> 2321 root 18 -2 55.8g 54g 51g S 47.7 28.6 0:07.75<br>
> AFPacketeth516<br>
> 2315 root 18 -2 55.8g 54g 51g R 45.2 28.6 0:12.18<br>
> AFPacketeth510<br>
> 2306 root 22 2 55.8g 54g 51g R 37.3 28.6 0:12.32<br>
> AFPacketeth51<br>
> 2312 root 18 -2 55.8g 54g 51g S 35.8 28.6 0:11.90<br>
> AFPacketeth57<br>
> 2308 root 20 0 55.8g 54g 51g R 34.8 28.6 0:16.69<br>
> AFPacketeth53<br>
> 2317 root 18 -2 55.8g 54g 51g R 33.3 28.6 0:07.93<br>
> AFPacketeth512<br>
> 2316 root 18 -2 55.8g 54g 51g S 28.8 28.6 0:08.03<br>
> AFPacketeth511<br>
> 2311 root 18 -2 55.8g 54g 51g S 24.9 28.6 0:10.51<br>
> AFPacketeth56<br>
> 2331 root 18 -2 55.8g 54g 51g R 19.9 28.6 0:02.41<br>
> AFPacketeth710<br>
> 2323 root 18 -2 55.8g 54g 51g S 17.9 28.6 0:03.60<br>
> AFPacketeth72<br>
> 2336 root 18 -2 55.8g 54g 51g S 16.9 28.6 0:01.50<br>
> AFPacketeth715<br>
> 2333 root 18 -2 55.8g 54g 51g S 14.9 28.6 0:02.14<br>
> AFPacketeth712<br>
> 2330 root 18 -2 55.8g 54g 51g S 13.9 28.6 0:02.12<br>
> AFPacketeth79<br>
> 2324 root 18 -2 55.8g 54g 51g R 11.9 28.6 0:02.96<br>
> AFPacketeth73<br>
> 2329 root 18 -2 55.8g 54g 51g S 11.9 28.6 0:01.90<br>
> AFPacketeth78<br>
> 2335 root 18 -2 55.8g 54g 51g S 11.9 28.6 0:01.44<br>
> AFPacketeth714<br>
> 2334 root 18 -2 55.8g 54g 51g R 10.9 28.6 0:01.68<br>
> AFPacketeth713<br>
> 2325 root 18 -2 55.8g 54g 51g S 9.4 28.6 0:02.38<br>
> AFPacketeth74<br>
> 2326 root 18 -2 55.8g 54g 51g S 8.9 28.6 0:02.71<br>
> AFPacketeth75<br>
> 2327 root 18 -2 55.8g 54g 51g S 7.5 28.6 0:01.98<br>
> AFPacketeth76<br>
> 2332 root 18 -2 55.8g 54g 51g S 7.5 28.6 0:01.53<br>
> AFPacketeth711<br>
> 2337 root 18 -2 55.8g 54g 51g S 7.0 28.6 0:01.09<br>
> AFPacketeth716<br>
> 2328 root 18 -2 55.8g 54g 51g S 6.0 28.6 0:02.11<br>
> AFPacketeth77<br>
> 2322 root 18 -2 55.8g 54g 51g R 5.5 28.6 0:03.78<br>
> AFPacketeth71<br>
> 3 root 20 0 0 0 0 S 4.5 0.0 0:01.25<br>
> ksoftirqd/0<br>
> 11 root 20 0 0 0 0 S 0.5 0.0 0:00.14<br>
> kworker/0:1<br>
><br>
> Regards<br>
><br>
><br>
> _______________________________________________<br>
> Suricata IDS Users mailing list:<br>
> <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support:<br>
> <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
> List:<br>
> <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
><br>
><br>
> Hi,<br>
><br>
><br>
> You could try "runmode: workers".<br>
<br>
</div></div>From thread name it seems it is already the case.<br>
<div><br>
><br>
><br>
> What is your flow balance method?<br>
><br>
> Can you try "flow per cpu" in the yaml section of afpacket?<br>
> ("cluster-type: cluster_cpu")<br>
<br>
</div>It could help indeed.<br>
<br>
A few questions:<br>
<br>
Are your IRQ affinity setting correct ? (meaning multiqueue used on the<br>
NICs and well balanced accross CPU ?)<br>
<br>
If you have a lot of UDP on your network use ethtool to load balance it<br>
as it is not done by default.<br>
<br>
BR,<br>
<div>><br>
><br>
><br>
><br>
><br>
> Thank you<br>
><br>
><br>
> --<br>
> Regards,<br>
> Peter Manev<br>
</div><div><div>> _______________________________________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
<br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>