<div dir="ltr"><div>For your question about the "high" profile setting over custom, at <a href="https://github.com/OISF/suricata/blob/8b87801b80f16d4b24c419221b49e43370ff6932/src/detect-engine.c#L2237">https://github.com/OISF/suricata/blob/8b87801b80f16d4b24c419221b49e43370ff6932/src/detect-engine.c#L2237</a> we can see the toclient-groups and toserver-groups values are each set to 75 in this case. It is possible the recommended settings of 200 from the docs would lower CPU usage even more, but for us high has worked well so thought I would let you know that is what we do. Raising those to 200 will likely increase rule load times even more so there is no reason for us to more than double these values when our performance is completely fine. I don't know a ton about the far reaching effects of these settings but I am guessing the trick is finding the right spot between memory and CPU usage without getting to the point where there is a a lot of overhead from large amounts of memory to manage.</div><div><br></div><div><div><div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;white-space:nowrap">-- </span></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;font-weight:bold;white-space:nowrap">Eric Urban</span><br></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap">University Information Security | Office of Information Technology | </span><a href="http://it.umn.edu/" style="color:rgb(17,85,204);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">it.umn.edu</a><br style="color:rgb(0,0,0);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap"><span style="color:rgb(0,0,0);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap">University of Minnesota | </span><a href="http://umn.edu/" style="color:rgb(17,85,204);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">umn.edu</a><br style="color:rgb(0,0,0);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap"><a href="mailto:eurban@umn.edu" style="color:rgb(17,85,204);font-family:'Helvetica Neue',Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">eurban@umn.edu</a><font style="color:rgb(136,136,136);font-size:12.8px" face="verdana, sans-serif"><br></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 20, 2019 at 12:56 AM Fabian Franz <<a href="mailto:fabfaeb@googlemail.com">fabfaeb@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Eric,</div><div>Hi Peter,<br></div><div><br></div><div>thanks for your feedback. It seems like I was able to resolve the issue with a combination of the steps you proposed:</div><div><ol><li>I changed the detect profile from the "recommended custom settings" to high,</li><li>switched the sgh-mpm-context to full,<br></li><li>set the max-pending-packets value back to 1024 from 8192, <br></li><li>took a closer look at septun and followed the cpu-affinity/core isolation steps and <br></li><li>switched up the testing setup a bit.</li></ol><div>Now I am facing a load of mostly 20-40% with one or two cores going up to 60% at 9Gbit traffic which seems fine to me. <br></div><div>Could you elaborate on why the "high" detect profile should be preferred over the custom settings recommended in the docs (<a href="https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html" target="_blank">https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html</a>) and why the lower max-pending-packets could have helped?<br></div><div>I still need to do some more testing on this and will come back to you if I have any new insights - but for now I think this is resolved.</div><div><br></div><div>Once again, thanks a lot for your fast response!</div><div><br></div><div>Best</div><div>Fabian<br></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Di., 20. Aug. 2019 um 01:04 Uhr schrieb Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><br><div dir="ltr">On 19 Aug 2019, at 12:06, Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>> wrote:<br><br></div><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div>We use Myricom cards with about 35K rules loaded. None of our cores run near 100% load. I saw one period of 6Gbps traffic in the last week on one of our Suricata instances where one core had 43% usage but the other 8 were at about 12%.<br></div><div><br></div><div>Have you looked at the Suricata Extreme Performance Tuning guide at <a href="https://github.com/pevma/SEPTun" target="_blank">https://github.com/pevma/SEPTun</a>? The cpu-affinity settings seem to be covered more in depth there than at the link that you posted. <br></div><div><br></div><div>Also, the section at <a href="https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html" target="_blank">https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html</a> could be of help. We don't use the custom setting recommended there but do use "high" for the profile and "full" for the sgh-mpm-context. Note the warning about significantly longer rule load times though.<br></div></div></div></blockquote><div><br></div><div>+1 for “high” context.</div><div><br></div><div>Fabian:</div><div>What is your max-pending packets value ?</div><div>Also in some tests/live set ups it is common to see some CPUs busier than others.</div><div><br></div><br><blockquote type="cite"><div dir="ltr"><div dir="ltr"><div><br></div><div><div><div dir="ltr" class="gmail-m_4639069194385070117gmail-m_-6978591599749042689gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;white-space:nowrap">-- </span></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;font-weight:bold;white-space:nowrap">Eric Urban</span><br></div><div dir="ltr"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap">University Information Security | Office of Information Technology | </span><a href="http://it.umn.edu/" style="color:rgb(17,85,204);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">it.umn.edu</a><br style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap"><span style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap">University of Minnesota | </span><a href="http://umn.edu/" style="color:rgb(17,85,204);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">umn.edu</a><br style="color:rgb(0,0,0);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap"><a href="mailto:eurban@umn.edu" style="color:rgb(17,85,204);font-family:"Helvetica Neue",Helvetica,sans-serif;font-size:small;line-height:17.29px;white-space:nowrap" target="_blank">eurban@umn.edu</a><font style="color:rgb(136,136,136);font-size:12.8px" face="verdana, sans-serif"><br></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 16, 2019 at 10:24 AM Fabian Franz <<a href="mailto:fabfaeb@googlemail.com" target="_blank">fabfaeb@googlemail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<p>Hi Everyone,</p>
<p>I am having a problem with my Suricata setup and hope that
someone here as a hint for me:<br>
I run suricata 4.1.4 together with a myricom card on a server with
128 gigs of RAM and two 16core(+HT) Intel CPUs.<br>
The SNF settings are 30 rings and 32/8gig for ringsizes. <br>
</p>
<p>As long as I do not deploy any rules, suricata runs smoothly with
~20% CPU load per (worker) core at 9-10 Gbit/s network traffic.
However, when I deploy even small rulesets (e.g. et-shellcode) the
CPU load skyrockets with 100% for 3-6 cores and the rest at around
50%. After a few moments, packets are dropped, with the SNF drop
ring full counter increasing rapidly (at 9-10Gbit/s, as before). I
use hyperscan as mpm-algo and tried to followed the
recommendations at<a href="https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/" target="_blank">
https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/ .</a><br>
However, I was not able to follow the recommendations regarding
IRQ, since those seemed pretty NIC specific. Is this setup also
relevant for myricom cards? <br>
Additionally, I obviously do not use AF_PACKET but libpcap with 30
threads. <br>
</p>
<p>To test the bandwidth I used iperf with 30 parallel connections.
Could this be the reason why only some of the cores are running at
100% load? If so, are there any other possiblities to simulate the
bandwidth more realistically?<br>
</p>
<p>Are there any myricom users here that could share performance
hints for myricom+suricata? I feel that (hardware-wise) my setup
should have no problem handling 10Gbit/s with a decent ruleset,
right?<br>
</p>
<p>Thanks a lot</p>
<p>Fabian<br>
</p>
</div>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a></blockquote></div>
</div></blockquote><blockquote type="cite"><div dir="ltr"><span>_______________________________________________</span><br><span>Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a></span><br><span>Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a></span><br><span>List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a></span><br><span></span><br><span>Conference: <a href="https://suricon.net" target="_blank">https://suricon.net</a></span><br><span>Trainings: <a href="https://suricata-ids.org/training/" target="_blank">https://suricata-ids.org/training/</a></span></div></blockquote></div></blockquote></div>
</blockquote></div>