[Oisf-users] Suricata, 10k rules, 10Gbit/sec and lots of RAM

[Peter Manev]

On Fri, Dec 4, 2015 at 1:56 AM, Michal Purzynski
<michalpurzynski1 at gmail.com> wrote:
> I kind of feel responsible here and should answer this question.
>
> The grouping code branch will make it to Suricata post 3.0. Give. The new release schedule, this should be quick.
>
> I'm testing it on production traffic, more than 20gbit, two sensors (peak, but frequent, long and crazy. Average is between 3 to 6gbit/sec).
>
> In order to stress the code I run it with even more insane settings, like this
>
> detect-engine:
>   - profile: custom
>   - custom-values:
>       toclient-src-groups: 2000
>       toclient-dst-groups: 2000
>       toclient-sp-groups: 2000
>       toclient-dp-groups: 3000
>       toserver-src-groups: 2000
>       toserver-dst-groups: 4000
>       toserver-sp-groups: 2000
>       toserver-dp-groups: 2500
>   - sgh-mpm-context: full
>   - inspection-recursion-limit: 3000
>   - rule-reload: true
>

I can confirm - steady performance improvement indeed based on all
observations so far.
Just to carify - the relevant config section in suricata.yaml for
branch  - https://github.com/inliniac/suricata/tree/dev-detect-grouping-v170
is:

detect-engine:
  - profile: custom
  - custom-values:
    toclient-groups: 50
    toserver-groups: 50

(please note the spelling)

And you  try 500, 1000 or more for the values.

Thank you



> Note - do not try this at home. Or work. It kills kittens on 2.x
>
> And it just works on the new branch that's yet to be merged :)
>
> Note - I have over 16500 rules now.
>
>> On 03 Dec 2015, at 23:12, Peter Manev <petermanev at gmail.com> wrote:
>>
>>> On Thu, Dec 3, 2015 at 11:05 PM, Duane Howard <duane.security at gmail.com> wrote:
>>> So the 'related' bug is still open, and targeted to 'Soon' does this mean we
>>> should expect a rewrite or refactoring of how the grouping works in 3.0, or
>>> somewhere further down the line?
>>
>> There is work already done with regards to grouping -
>> https://github.com/inliniac/suricata/tree/dev-detect-grouping-v170
>> Feedback is welcome.
>>
>> With the tests so far - it shows very god performance improvement and
>> minimal impact on memory (as compared to the old one)
>> It will most likely be post 3.0.
>>
>>>
>>>> On Wed, Nov 5, 2014 at 7:33 AM, Yasha Zislin <coolyasha at hotmail.com> wrote:
>>>>
>>>> I had this issue as well. setting sgh-mpm-context to full and my 132gb of
>>>> RAM would disappear without suricata fully starting.
>>>> I assume if setting this to full would increase performance if you have
>>>> sufficient hardware.
>>>>
>>>> My ruleset is 20k rules. :)
>>>>
>>>>> Date: Wed, 5 Nov 2014 11:24:01 +0100
>>>>> From: petermanev at gmail.com
>>>>> To: lists at inliniac.net
>>>>> CC: oisf-users at lists.openinfosecfoundation.org
>>>>> Subject: Re: [Oisf-users] Suricata, 10k rules, 10Gbit/sec and lots of
>>>>> RAM
>>>>
>>>>>
>>>>> On Wed, Nov 5, 2014 at 10:28 AM, Victor Julien <lists at inliniac.net>
>>>>> wrote:
>>>>>> On 11/05/2014 08:11 AM, Peter Manev wrote:
>>>>>>>> I'm kind of concerned that rules cannot fit in the memory with
>>>>>>>>> sgh-mpm-context set to full and the settings presented. Should I
>>>>>>>>> be?
>>>>>>>>> :)
>>>>>>> There is a bug at the moment when using full with over 10k rules - it
>>>>>>> just ends up eating all the memory.
>>>>>>
>>>>>> What bug is this?
>>>>>
>>>>> Tightly related to -
>>>>> https://redmine.openinfosecfoundation.org/issues/1202#change-4344
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Peter Manev
>>>>> _______________________________________________
>>>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>>> Site: http://suricata-ids.org | Support:
>>>>> http://suricata-ids.org/support/
>>>>> List:
>>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>> Training now available: http://suricata-ids.org/training/
>>>>
>>>> _______________________________________________
>>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>> Training now available: http://suricata-ids.org/training/
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net



-- 
Regards,
Peter Manev