[Oisf-users] Suricata, 10k rules, 10Gbit/sec and lots of RAM

[Victor Julien]

On 08-12-15 19:07, Cooper F. Nelson wrote:
> Performance is better, but not overwhelmingly so.  We can now run
> the full ETPRO subscription with mostly double-digit idle times
> (as displayed in top).  This was a long-time goal of mine.  Drop
> rate is identical (under .5% over a week).

Good news. The goal of this rewrite was not to get more performance,
but to get a code base I could understand again ;)

I was mostly worried about heavily tuned users like yourself getting
hurt by this somehow, so I'm happy to hear this isn't the case.

> If you wanted a scientific analysis I would think the right thing
> to do would be to record some traffic and then run it in offline
> mode with the performance counters enabled.  Unfortunately this
> isn't something we can do in our current configuration.

I have such a setup, and it does show a similar improvement. It's a
limited test of course, which is why I'm hoping for feedback
especially from the power users.

> Memory use is a little higher, however we are running a somewhat
> unique configuration regarding flow tracking.  %MEM does seem to be
> growing slightly over time, by about 1% a day.
> 
> As an aside, I saw this on one of the programming boards I monitor.
> Do you think it would be possible to implement any of these
> techniques within suricata's Boyer-Moore implementation (assuming
> they aren't already)?
> 
>> https://lists.freebsd.org/pipermail/freebsd-current/2010-August/019310.html

Our
>> 
main performance hit in the multi pattern matching (mpm) stage.
We've used a skip based algorithm in the past (b2g is still in our
tree), but performance with AC is quite a lot better. Generally the
problem for IDS patterns is that they are of poor quality, many 1 and
2 byte patterns. These defeat the skip based algo's. Another issue
that is important to us is the worst-case performance. The skip based
algo's seem to have a worse worst case profile.

Btw, I recently saw a new paper on a mix of AC and skip based approach
that I still have to take a deeper look at:
http://halcyon.usc.edu/~pk/prasannawebsite/papers/HeadBody_camera.pdf

Finally, we should start experimenting with Intel's Hyperscan soon.
They claim much better perf, so we will see :)

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------