[Oisf-users] Loss of output without visual loss in stats.log.

[Andreas Moe]

Hi there.

This is mostly just a initial query into if any of you other users have
experienced any similar scenarios as i have during some testing. This has
earlier been presented to members of the Suricata dev-team, but this round
we are bringing more data, charts and looking to find out exactly what is
going on.

Before asking, we have performed all tuning, offloading and profiling
(suricata, kernel and drivers) based on best-practices found on the
internett (like Regit, Pevma, tips from Victor, and so on) and also our own
work and better understanding of suricata and related systems over the past
year.

System outline:
CentOS 6, 128GB RAM, 40 Cores, Both fiber 10Gbit/s and copper 1Gbit/s.

Testing outline:
PCAPs captured at different speeds (slow, 1-5Mbit/s, and up towards
800Mbit/s). PCAPs are replayed multiple times at different speeds
(100,250,500,750, and topspeed setting of tcpreplay). Data varies from
malwareactivity, scanning, normal legitimat use, etc. Also used varying
rulesets containing anything from 5000 to 25000 signatures.

What we measured:
Looked at the stats log file to look for drops, reassembly gaps,
fragmentation, memcaps, all of that.

Results outline:
The higher the speed (Mbit/s) even with stats log indicating little or no
drop, no missing reassembly, no reaced memcaps and so on, output is
missing. With output here i mean say, alerts (unified2, fastlog, or EVE),
and the same goes for http/dns log entries (either the old format or EVE).

Example #1: At 100mbit to 750 mbit we see a drastic decrase of http log
entries (many thousand mising).

Example #2: Without pcap-log at 100Mbit/s compared to with pcap-log at
750Mbit/s we are missing over 80% of many log entries.

Ofcourse, there could be something i have done wrong, but with all the
tuning, testing and experience with Suricata i cant see what else it can
be. Has anyone else performed similar testing and or seen similar results?

Sincerly, Andreas Moe.

P.s. I know that the presented data is very vauge, but as mentiond this is
just an initial inquery.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150605/5a7a7b84/attachment.html>