[Oisf-users] Suricata, 10k rules, 10Gbit/sec and lots of RAM

[Michal Purzynski]

I kind of feel responsible here and should answer this question.

The grouping code branch will make it to Suricata post 3.0. Give. The new release schedule, this should be quick.

I'm testing it on production traffic, more than 20gbit, two sensors (peak, but frequent, long and crazy. Average is between 3 to 6gbit/sec).

In order to stress the code I run it with even more insane settings, like this

detect-engine:
  - profile: custom
  - custom-values:
      toclient-src-groups: 2000
      toclient-dst-groups: 2000
      toclient-sp-groups: 2000
      toclient-dp-groups: 3000
      toserver-src-groups: 2000
      toserver-dst-groups: 4000
      toserver-sp-groups: 2000
      toserver-dp-groups: 2500
  - sgh-mpm-context: full
  - inspection-recursion-limit: 3000
  - rule-reload: true

Note - do not try this at home. Or work. It kills kittens on 2.x

And it just works on the new branch that's yet to be merged :)

Note - I have over 16500 rules now.

> On 03 Dec 2015, at 23:12, Peter Manev <petermanev at gmail.com> wrote:
> 
>> On Thu, Dec 3, 2015 at 11:05 PM, Duane Howard <duane.security at gmail.com> wrote:
>> So the 'related' bug is still open, and targeted to 'Soon' does this mean we
>> should expect a rewrite or refactoring of how the grouping works in 3.0, or
>> somewhere further down the line?
> 
> There is work already done with regards to grouping -
> https://github.com/inliniac/suricata/tree/dev-detect-grouping-v170
> Feedback is welcome.
> 
> With the tests so far - it shows very god performance improvement and
> minimal impact on memory (as compared to the old one)
> It will most likely be post 3.0.
> 
>> 
>>> On Wed, Nov 5, 2014 at 7:33 AM, Yasha Zislin <coolyasha at hotmail.com> wrote:
>>> 
>>> I had this issue as well. setting sgh-mpm-context to full and my 132gb of
>>> RAM would disappear without suricata fully starting.
>>> I assume if setting this to full would increase performance if you have
>>> sufficient hardware.
>>> 
>>> My ruleset is 20k rules. :)
>>> 
>>>> Date: Wed, 5 Nov 2014 11:24:01 +0100
>>>> From: petermanev at gmail.com
>>>> To: lists at inliniac.net
>>>> CC: oisf-users at lists.openinfosecfoundation.org
>>>> Subject: Re: [Oisf-users] Suricata, 10k rules, 10Gbit/sec and lots of
>>>> RAM
>>> 
>>>> 
>>>> On Wed, Nov 5, 2014 at 10:28 AM, Victor Julien <lists at inliniac.net>
>>>> wrote:
>>>>> On 11/05/2014 08:11 AM, Peter Manev wrote:
>>>>>>> I'm kind of concerned that rules cannot fit in the memory with
>>>>>>>> sgh-mpm-context set to full and the settings presented. Should I
>>>>>>>> be?
>>>>>>>> :)
>>>>>> There is a bug at the moment when using full with over 10k rules - it
>>>>>> just ends up eating all the memory.
>>>>> 
>>>>> What bug is this?
>>>> 
>>>> Tightly related to -
>>>> https://redmine.openinfosecfoundation.org/issues/1202#change-4344
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Regards,
>>>> Peter Manev
>>>> _______________________________________________
>>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>>> Site: http://suricata-ids.org | Support:
>>>> http://suricata-ids.org/support/
>>>> List:
>>>> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>> Training now available: http://suricata-ids.org/training/
>>> 
>>> _______________________________________________
>>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>> Training now available: http://suricata-ids.org/training/
> 
> 
> 
> -- 
> Regards,
> Peter Manev
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net