[Oisf-users] Unbalanced load on AFpacket threads
Fernando Sclavo
fsclavo at gmail.com
Mon Jun 3 13:34:19 UTC 2013
Hi all!
We are running Suricata 1.4.2 with two Intel x520 cards, connected each one
to the core switches on our datacenter network. The average traffic is
about 1~2Gbps per port.
As you can see on the following top output, there are some threads
significantly more loaded than others (AFPacketeth54 for example): these
threads are continuously dropping kernel packets. We raised kernel
parameters (buffers and rmem, etc) and lowered suricata timeouts flows to
just a few seconds, but we can't keep drops counter static when CPU goes to
99.9% for a specific thread.
How can we do to balance the load better on all threads to prevent this
issue?
The server is a Dell R715 2x16 core AMD Opteron(tm) Processor 6284, 192Gb
RAM.
idsuser at suricata:~$ top -d2
top - 10:24:05 up 1 min, 2 users, load average: 4.49, 1.14, 0.38
Tasks: 287 total, 15 running, 272 sleeping, 0 stopped, 0 zombie
Cpu(s): 30.3%us, 1.3%sy, 0.0%ni, 65.3%id, 0.0%wa, 0.0%hi, 3.1%si,
0.0%st
Mem: 198002932k total, 59619020k used, 138383912k free, 25644k buffers
Swap: 15624188k total, 0k used, 15624188k free, 161068k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2309 root 18 -2 55.8g 54g 51g R 99.9 28.6 0:20.96
AFPacketeth54
2314 root 18 -2 55.8g 54g 51g R 99.9 28.6 0:18.29
AFPacketeth59
2318 root 18 -2 55.8g 54g 51g R 99.9 28.6 0:12.90
AFPacketeth513
2319 root 18 -2 55.8g 54g 51g R 77.6 28.6 0:12.78
AFPacketeth514
2307 root 20 0 55.8g 54g 51g S 66.6 28.6 0:21.25
AFPacketeth52
2338 root 20 0 55.8g 54g 51g R 58.2 28.6 0:09.94
FlowManagerThre
2310 root 18 -2 55.8g 54g 51g S 51.2 28.6 0:15.35
AFPacketeth55
2320 root 18 -2 55.8g 54g 51g R 50.2 28.6 0:07.83
AFPacketeth515
2313 root 18 -2 55.8g 54g 51g S 48.7 28.6 0:11.66
AFPacketeth58
2321 root 18 -2 55.8g 54g 51g S 47.7 28.6 0:07.75
AFPacketeth516
2315 root 18 -2 55.8g 54g 51g R 45.2 28.6 0:12.18
AFPacketeth510
2306 root 22 2 55.8g 54g 51g R 37.3 28.6 0:12.32
AFPacketeth51
2312 root 18 -2 55.8g 54g 51g S 35.8 28.6 0:11.90
AFPacketeth57
2308 root 20 0 55.8g 54g 51g R 34.8 28.6 0:16.69
AFPacketeth53
2317 root 18 -2 55.8g 54g 51g R 33.3 28.6 0:07.93
AFPacketeth512
2316 root 18 -2 55.8g 54g 51g S 28.8 28.6 0:08.03
AFPacketeth511
2311 root 18 -2 55.8g 54g 51g S 24.9 28.6 0:10.51
AFPacketeth56
2331 root 18 -2 55.8g 54g 51g R 19.9 28.6 0:02.41
AFPacketeth710
2323 root 18 -2 55.8g 54g 51g S 17.9 28.6 0:03.60
AFPacketeth72
2336 root 18 -2 55.8g 54g 51g S 16.9 28.6 0:01.50
AFPacketeth715
2333 root 18 -2 55.8g 54g 51g S 14.9 28.6 0:02.14
AFPacketeth712
2330 root 18 -2 55.8g 54g 51g S 13.9 28.6 0:02.12
AFPacketeth79
2324 root 18 -2 55.8g 54g 51g R 11.9 28.6 0:02.96
AFPacketeth73
2329 root 18 -2 55.8g 54g 51g S 11.9 28.6 0:01.90
AFPacketeth78
2335 root 18 -2 55.8g 54g 51g S 11.9 28.6 0:01.44
AFPacketeth714
2334 root 18 -2 55.8g 54g 51g R 10.9 28.6 0:01.68
AFPacketeth713
2325 root 18 -2 55.8g 54g 51g S 9.4 28.6 0:02.38
AFPacketeth74
2326 root 18 -2 55.8g 54g 51g S 8.9 28.6 0:02.71
AFPacketeth75
2327 root 18 -2 55.8g 54g 51g S 7.5 28.6 0:01.98
AFPacketeth76
2332 root 18 -2 55.8g 54g 51g S 7.5 28.6 0:01.53
AFPacketeth711
2337 root 18 -2 55.8g 54g 51g S 7.0 28.6 0:01.09
AFPacketeth716
2328 root 18 -2 55.8g 54g 51g S 6.0 28.6 0:02.11
AFPacketeth77
2322 root 18 -2 55.8g 54g 51g R 5.5 28.6 0:03.78
AFPacketeth71
3 root 20 0 0 0 0 S 4.5 0.0 0:01.25
ksoftirqd/0
11 root 20 0 0 0 0 S 0.5 0.0 0:00.14
kworker/0:1
Regards
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20130603/af14e0de/attachment.html>
More information about the Oisf-users
mailing list