[Oisf-users] High CPU Load with Small Ruleset at 10Gbit/s

[Fabian Franz]

Hi Eric,
Hi Peter,

thanks for your feedback. It seems like I was able to resolve the issue
with a combination of the steps you proposed:

   1. I changed the detect profile from the "recommended custom settings"
   to high,
   2. switched the sgh-mpm-context to full,
   3. set the max-pending-packets value back to 1024 from 8192,
   4. took a closer look at septun and followed the cpu-affinity/core
   isolation steps and
   5. switched up the testing setup a bit.

Now I am facing a load of mostly 20-40% with one or two cores going up to
60% at 9Gbit traffic which seems fine to me.
Could you elaborate on why the "high" detect profile should be preferred
over the custom settings recommended in the docs (
https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html)
and why the lower max-pending-packets could have helped?
I still need to do some more testing on this and will come back to you if I
have any new insights - but for now I think this is resolved.

Once again, thanks a lot for your fast response!

Best
Fabian

Am Di., 20. Aug. 2019 um 01:04 Uhr schrieb Peter Manev <petermanev at gmail.com
>:

>
> On 19 Aug 2019, at 12:06, Eric Urban <eurban at umn.edu> wrote:
>
> We use Myricom cards with about 35K rules loaded.  None of our cores run
> near 100% load.  I saw one period of 6Gbps traffic in the last week on one
> of our Suricata instances where one core had 43% usage but the other 8 were
> at about 12%.
>
> Have you looked at the Suricata Extreme Performance Tuning guide at
> https://github.com/pevma/SEPTun?  The cpu-affinity settings seem to be
> covered more in depth there than at the link that you posted.
>
> Also, the section at
> https://suricata.readthedocs.io/en/latest/performance/high-performance-config.html
> could be of help.  We don't use the custom setting recommended there but do
> use "high" for the profile and "full" for the sgh-mpm-context.  Note the
> warning about significantly longer rule load times though.
>
>
> +1 for “high” context.
>
> Fabian:
> What is your max-pending packets value ?
> Also in some tests/live set ups it is common to see some CPUs busier than
> others.
>
>
>
> --
> Eric Urban
> University Information Security | Office of Information Technology |
> it.umn.edu
> University of Minnesota | umn.edu
> eurban at umn.edu
>
>
> On Fri, Aug 16, 2019 at 10:24 AM Fabian Franz <fabfaeb at googlemail.com>
> wrote:
>
>> Hi Everyone,
>>
>> I am having a problem with my Suricata setup and hope that someone here
>> as a hint for me:
>> I run suricata 4.1.4 together with a myricom card on a server with 128
>> gigs of RAM and two 16core(+HT) Intel CPUs.
>> The SNF settings are 30 rings and 32/8gig for ringsizes.
>>
>> As long as I do not deploy any rules, suricata runs smoothly with ~20%
>> CPU load per (worker) core at 9-10 Gbit/s network traffic. However, when I
>> deploy even small rulesets (e.g. et-shellcode) the CPU load skyrockets with
>> 100% for 3-6 cores and the rest at around 50%. After a few moments, packets
>> are dropped, with the SNF drop ring full counter increasing rapidly (at
>> 9-10Gbit/s, as before). I use hyperscan as mpm-algo and tried to followed
>> the recommendations at
>> https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/ .
>> <https://home.regit.org/2012/07/suricata-to-10gbps-and-beyond/>
>> However, I was not able to follow the recommendations regarding IRQ,
>> since those seemed pretty NIC specific. Is this setup also relevant for
>> myricom cards?
>> Additionally, I obviously do not use AF_PACKET but libpcap with 30
>> threads.
>>
>> To test the bandwidth I used iperf with 30 parallel connections. Could
>> this be the reason why only some of the cores are running at 100% load? If
>> so, are there any other possiblities to simulate the bandwidth more
>> realistically?
>>
>> Are there any myricom users here that could share performance hints for
>> myricom+suricata? I feel that (hardware-wise) my setup should have no
>> problem handling 10Gbit/s with a decent ruleset, right?
>>
>> Thanks a lot
>>
>> Fabian
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>> Conference: https://suricon.net
>> Trainings: https://suricata-ids.org/training/
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190820/07748d97/attachment-0001.html>