[Oisf-users] Suricata, 10k rules, 10Gbit/sec and lots of RAM

[Peter Manev]

On Wed, Dec 9, 2015 at 2:36 PM, Victor Julien <lists at inliniac.net> wrote:
> On 08-12-15 19:07, Cooper F. Nelson wrote:
>> Performance is better, but not overwhelmingly so.  We can now run
>> the full ETPRO subscription with mostly double-digit idle times
>> (as displayed in top).  This was a long-time goal of mine.  Drop
>> rate is identical (under .5% over a week).
>
> Good news. The goal of this rewrite was not to get more performance,
> but to get a code base I could understand again ;)
>
> I was mostly worried about heavily tuned users like yourself getting
> hurt by this somehow, so I'm happy to hear this isn't the case.


We should not forget that there are different(other) use cases where
that particular git branch is doing quite an improvement.
Thanks to the work there - now it is possible to actually get a full
ET ruleset and have HOME_NET set up as "any"  - with minimal drops.

>
>> If you wanted a scientific analysis I would think the right thing
>> to do would be to record some traffic and then run it in offline
>> mode with the performance counters enabled.  Unfortunately this
>> isn't something we can do in our current configuration.
>
> I have such a setup, and it does show a similar improvement. It's a
> limited test of course, which is why I'm hoping for feedback
> especially from the power users.
>
>> Memory use is a little higher, however we are running a somewhat
>> unique configuration regarding flow tracking.  %MEM does seem to be
>> growing slightly over time, by about 1% a day.
>>
>> As an aside, I saw this on one of the programming boards I monitor.
>> Do you think it would be possible to implement any of these
>> techniques within suricata's Boyer-Moore implementation (assuming
>> they aren't already)?
>>
>>> https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html
>
> Our
>>>
> main performance hit in the multi pattern matching (mpm) stage.
> We've used a skip based algorithm in the past (b2g is still in our
> tree), but performance with AC is quite a lot better. Generally the
> problem for IDS patterns is that they are of poor quality, many 1 and
> 2 byte patterns. These defeat the skip based algo's. Another issue
> that is important to us is the worst-case performance. The skip based
> algo's seem to have a worse worst case profile.
>
> Btw, I recently saw a new paper on a mix of AC and skip based approach
> that I still have to take a deeper look at:
> http://halcyon.usc.edu/~pk/prasannawebsite/papers/HeadBody_camera.pdf
>
> Finally, we should start experimenting with Intel's Hyperscan soon.
> They claim much better perf, so we will see :)
>
> Cheers,
> Victor
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net



-- 
Regards,
Peter Manev