<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
</head>
<body dir="ltr">
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>top -H -p suricata_pid gets me this:</p>
<p></p>
<div><br>
</div>
<div><br>
</div>
<div> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND</div>
<div> 4692 suricata 20 0 11.2g 10g 396m R 100.2 69.5 69:24.27 W#03-bond0</div>
<div> 4693 suricata 20 0 11.2g 10g 396m S 52.6 69.5 55:26.68 W#04-bond0</div>
<div> 4690 suricata 20 0 11.2g 10g 396m R 45.6 69.5 67:32.78 W#01-bond0</div>
<div> 4691 suricata 20 0 11.2g 10g 396m S 30.6 69.5 55:52.85 W#02-bond0</div>
<div> 4694 suricata 20 0 11.2g 10g 396m S 1.3 69.5 1:20.04 FM#01</div>
<div> 4695 suricata 20 0 11.2g 10g 396m S 0.7 69.5 0:32.82 FM#02</div>
<div> 4442 suricata 20 0 11.2g 10g 396m S 0.3 69.5 3:45.30 Suricata-Main</div>
<div> 4697 suricata 20 0 11.2g 10g 396m S 0.0 69.5 0:08.25 FR#01</div>
<div> 4698 suricata 20 0 11.2g 10g 396m S 0.0 69.5 0:00.05 CW</div>
<div> 4699 suricata 20 0 11.2g 10g 396m S 0.0 69.5 0:00.15 CS</div>
<div><br>
</div>
W#04 detection threads gets higher CPU utilization as well but not 100%. It seems that it is unevenly spread.
<p></p>
<p>So the reason that it says bond0 is because I've bonded two nics into one since they are taps and each sees one direction of traffic.</p>
<p><br>
</p>
<p>Looking at cat /proc/interrupts | grep eth0[2]</p>
<p></p>
<div>56: 1499740917 0 1 12313392 PCI-MSI-edge eth0</div>
<div> 57: 0 0 0 0 PCI-MSI-edge eth0:1</div>
<div> 58: 0 0 0 0 PCI-MSI-edge eth0:2</div>
<div> 59: 0 0 0 0 PCI-MSI-edge eth0:3</div>
<div> 64: 0 0 2 3682766018 PCI-MSI-edge eth2</div>
<div> 65: 0 0 0 0 PCI-MSI-edge eth2:1</div>
<div> 66: 0 0 0 0 PCI-MSI-edge eth2:2</div>
<div> 67: 0 0 0 0 PCI-MSI-edge eth2:3</div>
<div><br>
</div>
It seems that each NIC is only utilizing one queue and one CPU. Could this explain HIGH CPU usage on two detection threads?
<p></p>
<p>I've tried to assign smp affinity to all nic queues for all CPUs but it didnt seem to change much in these stats above. And yes I've killed irqbalance when I've tested that.</p>
<p><br>
</p>
<p>After running perf top (without -c option), the highest usage is </p>
<p></p>
<div> 6.15% libc-2.12.so [.] __memset_x86_64</div>
<div><br>
</div>
Running perf top -c 0, I get the same results.
<p></p>
<p><br>
</p>
<p>Running perf top -c 1 or 2 or 3, I get only one line (sometimes two)</p>
<p></p>
<div> 100.00% [kernel] [k] native_write_msr_safe</div>
<div><br>
</div>
<div>or </div>
<div><br>
</div>
<div>
<div>
<div> 77.00% [kernel] [k] native_write_msr_safe</div>
<div> 23.00% libc-2.12.so [.] memcpy</div>
<div><br>
</div>
<div>BTW, i get weird numbers for symbol under suricata processes in perf top. I've found an article about that which tells to compile with <code style="margin: 0px; padding: 1px 5px; border: 0px; font-size: 13px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; white-space: pre-wrap; color: rgb(36, 39, 41); background-color: rgb(239, 240, 241);">-fno-omit-frame-pointer
flag for GCC. No help.</code></div>
<div><code style="margin: 0px; padding: 1px 5px; border: 0px; font-size: 13px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; white-space: pre-wrap; color: rgb(36, 39, 41); background-color: rgb(239, 240, 241);"><br>
</code></div>
<div><code style="margin: 0px; padding: 1px 5px; border: 0px; font-size: 13px; font-family: Consolas, Menlo, Monaco, "Lucida Console", "Liberation Mono", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Courier New", monospace, sans-serif; white-space: pre-wrap; color: rgb(36, 39, 41); background-color: rgb(239, 240, 241);">Thanks.</code></div>
<div><br>
</div>
</div>
<div><br>
</div>
<br>
</div>
<br>
<p></p>
<p><br>
</p>
<br>
<br>
<div style="color: rgb(0, 0, 0);">
<div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Peter Manev <petermanev@gmail.com><br>
<b>Sent:</b> Thursday, June 30, 2016 4:27 PM<br>
<b>To:</b> Yasha Zislin<br>
<b>Cc:</b> oisf-users@lists.openinfosecfoundation.org<br>
<b>Subject:</b> Re: [Oisf-users] number of alerts versus performance</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">On Thu, 2016-06-30 at 15:54 +0000, Yasha Zislin wrote:<br>
> Peter,<br>
> <br>
> <br>
> I found one alert that was causing high alert count. After I've<br>
> disabled it, count went down but packet loss is still around 20%.<br>
> <br>
> <br>
> my stats.log does not contain anything useful such as flow emergency<br>
> mode, or ssn memcap drop. The only thing that is off is kernel drops,<br>
> and tcp reassembly gaps. <br>
> From my understanding kernel drops have nothing to do with Suricata<br>
> and point to OS problems.<br>
> <br>
> <br>
> I do see one of the CPUs peak at 100% when packet loss increases. One<br>
> thing to note. Two other CPUs are working on capturing traffic with<br>
> high IRQs. My guess would be flow manager or detection engine.<br>
> <br>
<br>
<br>
You can see if you get more info from:<br>
top -H -p `pidof suricata`<br>
and<br>
perf top -c cpu_number_here<br>
example: perf top -c 0<br>
<br>
> I dunno.<br>
> <br>
> <br>
> Thanks<br>
> <br>
> <br>
> <br>
> <br>
> ______________________________________________________________________<br>
> From: Peter Manev <petermanev@gmail.com><br>
> Sent: Thursday, June 30, 2016 3:00 PM<br>
> To: Yasha Zislin<br>
> Cc: oisf-users@lists.openinfosecfoundation.org<br>
> Subject: Re: [Oisf-users] number of alerts versus performance <br>
> <br>
> On Thu, 2016-06-30 at 14:41 +0000, Yasha Zislin wrote:<br>
> > I have been trying to figure out a packet loss on one of my sensors<br>
> > and I am puzzled.<br>
> > <br>
> > It is has 16 gigs of RAM, one quad core AMD CPU, and nic sees about<br>
> 3<br>
> > million packets per minute. Nothing special in my mind. I am using<br>
> > PFRING 6.5.0 and Suricata 3.1.<br>
> > <br>
> > I get about 20% to 40% packet loss. I have another identical server<br>
> > which sees the same amount of traffic and maybe some of the same<br>
> > traffic as well.<br>
> > <br>
> > I've been messing around with NIC settings, IRQs, PFRING settings,<br>
> > Suricata settings trying to figure out why such a high packet loss.<br>
> > <br>
> > <br>
> > I have just realized one big difference in these two sensors.<br>
> > Problematic one gets 2k to 4k of alerts per minute which sounds<br>
> huge.<br>
> > <br>
> <br>
> Any particular sig that is alerting in excess ?<br>
> <br>
> > Second one gets like 80 alerts per minute. Both have the same<br>
> > rulesets.<br>
> > <br>
> > <br>
> > The difference of course is the home_net variable.<br>
> > <br>
> > <br>
> > Can the fact that Suricata processes more rules due to HOME_NET<br>
> > definition cause high performance strain on the server? <br>
> > <br>
> <br>
> Yes HOME_NET size has effect on performance as well (among other<br>
> things). For example - <br>
> HOME_NET: "any"<br>
> EXTERNAL_NET: "any"<br>
> will certainly degrade your performance.<br>
> <br>
> > <br>
> > If the packet does not match per HOME_NET, it will be discarded<br>
> before<br>
> > being processed in rules. Correct?<br>
> > <br>
> > Versus if packet passes HOME_NET check, it would have to go through<br>
> > all of the rules, hence cause higher CPU utilization.<br>
> > <br>
> > <br>
> > Thank you for the clarification.<br>
> > <br>
> > <br>
> > _______________________________________________<br>
> > Suricata IDS Users mailing list:<br>
> oisf-users@openinfosecfoundation.org<br>
> > Site: <a href="http://suricata-ids.org" id="LPlnk659662">http://suricata-ids.org</a> | Support:<br>
> <a href="http://suricata-ids.org/support/">http://suricata-ids.org/support/</a>
<br>
> <br>
> <br>
> Suricata<br>
> suricata-ids.org<br>
> Open Source IDS / IPS / NSM engine<br>
> <br>
> <br>
> > List:<br>
> <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> > Suricata User Conference November 9-11 in Washington, DC:<br>
> <a href="http://oisfevents.net">http://oisfevents.net</a><br>
> <br>
> <br>
<br>
-- <br>
Regards,<br>
Peter Manev<br>
<br>
</div>
</span></font></div>
</div>
</body>
</html>