<div dir="ltr"><div>Hi Eric,</div><div>Hi Peter,<br></div><div><br></div><div>thanks for your feedback. It seems like I was able to resolve the issue with a combination of the steps you proposed:</div><div><ol><li>I changed the detect profile from the "recommended custom settings" to high,</li><li>switched the sgh-mpm-context to full,<br></li><li>set the max-pending-packets value back to 1024 from 8192, <br></li><li>took a closer look at septun and followed the cpu-affinity/core isolation steps and <br></li><li>switched up the testing setup a bit.</li></ol><div>Now I am facing a load of mostly 20-40% with one or two cores going up to 60% at 9Gbit traffic which seems fine to me. <br></div><div>Could you elaborate on why the "high" detect profile should be preferred over the custom settings recommended in the docs (<a href="https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html">https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html</a>) and why the lower max-pending-packets could have helped?<br></div><div>I still need to do some more testing on this and will come back to you if I have any new insights - but for now I think this is resolved.</div><div><br></div><div>Once again, thanks a lot for your fast response!</div><div><br></div><div>Best</div><div>Fabian<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Di., 20. Aug. 2019 um 01:04 Uhr schrieb Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><br><div dir="ltr">On 19 Aug 2019, at 12:06, Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div>We use Myricom cards with about 35K rules loaded. None of our cores run near 100% load. I saw one period of 6Gbps traffic in the last week on one of our Suricata instances where one core had 43% usage but the other 8 were at about 12%.<br></div><div><br></div><div>Have you looked at the Suricata Extreme Performance Tuning guide at <a href="https://github.com/pevma/SEPTun" target="_blank">https://github.com/pevma/SEPTun</a>? The cpu-affinity settings seem to be covered more in depth there than at the link that you posted. <br></div><div><br></div><div>Also, the section at <a href="https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html" target="_blank">https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html</a> could be of help. We don't use the custom setting recommended there but do use "high" for the profile and "full" for the sgh-mpm-context. Note the warning about significantly longer rule load times though.<br></div></div></div></blockquote><div><br></div><div>+1 for “high” context.</div><div><br></div><div>Fabian:</div><div>What is your max-pending packets value ?</div><div>Also in some tests/live set ups it is common to see some CPUs busier than others.</div><div><br></div><br><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div><br></div><div><div><div dir="ltr" class="gmail-m_-6978591599749042689gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;white-space:nowrap">-- </span></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;font-weight:bold;white-space:nowrap">Eric Urban</span><br></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap">University Information Security | Office of Information Technology | </span><a href="http://it.umn.edu/" style="color:rgb(17,85,204);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">it.umn.edu</a><br style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap">University of Minnesota | </span><a href="http://umn.edu/" style="color:rgb(17,85,204);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">umn.edu</a><br style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap"><a href="mailto:eurban@umn.edu" style="color:rgb(17,85,204);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">eurban@umn.edu</a><font style="color:rgb(136,136,136);font-size:12.8px" face="verdana, sans-serif"><br></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 16, 2019 at 10:24 AM Fabian Franz <<a href="mailto:fabfaeb@googlemail.com" target="_blank">fabfaeb@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Everyone,</p>
<p>I am having a problem with my Suricata setup and hope that
someone here as a hint for me:<br>
I run suricata 4.1.4 together with a myricom card on a server with
128 gigs of RAM and two 16core(+HT) Intel CPUs.<br>
The SNF settings are 30 rings and 32/8gig for ringsizes. <br>
</p>
<p>As long as I do not deploy any rules, suricata runs smoothly with
~20% CPU load per (worker) core at 9-10 Gbit/s network traffic.
However, when I deploy even small rulesets (e.g. et-shellcode) the
CPU load skyrockets with 100% for 3-6 cores and the rest at around
50%. After a few moments, packets are dropped, with the SNF drop
ring full counter increasing rapidly (at 9-10Gbit/s, as before). I
use hyperscan as mpm-algo and tried to followed the
recommendations at<a href="https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/" target="_blank">
https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/ .</a><br>
However, I was not able to follow the recommendations regarding
IRQ, since those seemed pretty NIC specific. Is this setup also
relevant for myricom cards? <br>
Additionally, I obviously do not use AF_PACKET but libpcap with 30
threads. <br>
</p>
<p>To test the bandwidth I used iperf with 30 parallel connections.
Could this be the reason why only some of the cores are running at
100% load? If so, are there any other possiblities to simulate the
bandwidth more realistically?<br>
</p>
<p>Are there any myricom users here that could share performance
hints for myricom+suricata? I feel that (hardware-wise) my setup
should have no problem handling 10Gbit/s with a decent ruleset,
right?<br>
</p>
<p>Thanks a lot</p>
<p>Fabian<br>
</p>
</div>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a></blockquote></div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a></span><br><span>Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a></span><br><span>List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a></span><br><span></span><br><span>Conference: <a href="https://suricon.net" target="_blank">https://suricon.net</a></span><br><span>Trainings: <a href="https://suricata-ids.org/training/" target="_blank">https://suricata-ids.org/training/</a></span></div></blockquote></div></blockquote></div>