[Oisf-users] Testing

[Will Metcalf]

On Thu, Jan 14, 2010 at 11:14 AM, Al MailingList <
alpal.mailinglist at gmail.com> wrote:

> Hey guys,
>
> I notice some talk about people looking at replacing their production
> SNORT installs with Suricata installs?
>

 The current release is beta quality code.  I wouldn't go replacing any
production sensors until we have a production quality release.  I'm not
saying that the code isn't fairly stable, just that we have some things to
do performance tuning wise, the addition of some keywords etc. With all of
that said we are looking for willing beta testers to provide us feedback :D.

Just curious about the level of testing that's been done to ensure
> 100% true pos rates on detection etc?
>

If you mean are we a 100% drop-in replacement for snort at this point? No we
are not, but we are getting closer ;-).  For the functionality we have
implemented, we have nightly QA runs  with real traffic etc to test out
various aspects of the engine.  We also have unit tests built into the code
to ensure functionality and to detect regressions.  We run fuzzers around
the clock to find all kinds of interesting corner cases that may make the
engine explode etc.  Additionally we will never support the snort .so rules
as these rely on the snort engine itself.


> I still haven't had a chance to actually look at the code, but in
> particular interested in performance of the stream reassembler etc in
> real world scenarios? Has anyone done a bake off against SNORT with
> the same rules on realistic data rates?


Like I mentioned previously we still have some performance tuning to do.
But in the interest of full-disclosure we have gotten some reports that the
engine performs much better than snort and some reports that it performs
much worse. I have a feeling at this point this greatly depends on traffic
mixture and rules.  For example if you are reading from pcap the current run
mode for reading pcap files really doesn't take full advantage of threading
framework if you have more than two cores (someday soon the number of
threads for stream handling, detection, etc will be user configurable, and
or auto-tuned). Additionally currently we look at the entire http response
body whereas by default snort only looks at the first 300 bytes, this will
add additional overhead to suricata.  Also testing IDP accurately is hard
especially at high speeds, and it is also hard to perform an apples to
apples comparison.  Probably about the closest thing that we could get is
multiple flow-pinned snort processes but then certain things begin to break
down like thresholding as the snort processes no longer share this data
between them, so detection rates will differ here.

If anybody is setup to do testing like this with Breaking Point, Spirent, or
similar types of test boxes and would be willing to share their results with
the team and or with the public we would be ecstatic to have this sort of
feedback.

Regards,

Will

>
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20100117/666957be/attachment-0002.html>