[Oisf-users] Loss of output without visual loss in stats.log.

[Victor Julien]

On 06/05/2015 07:41 PM, Andreas Moe wrote:
> Hi there.
> 
> This is mostly just a initial query into if any of you other users have
> experienced any similar scenarios as i have during some testing. This
> has earlier been presented to members of the Suricata dev-team, but this
> round we are bringing more data, charts and looking to find out exactly
> what is going on.
> 
> Before asking, we have performed all tuning, offloading and profiling
> (suricata, kernel and drivers) based on best-practices found on the
> internett (like Regit, Pevma, tips from Victor, and so on) and also our
> own work and better understanding of suricata and related systems over
> the past year.
> 
> System outline:
> CentOS 6, 128GB RAM, 40 Cores, Both fiber 10Gbit/s and copper 1Gbit/s.
> 
> Testing outline:
> PCAPs captured at different speeds (slow, 1-5Mbit/s, and up towards
> 800Mbit/s). PCAPs are replayed multiple times at different speeds
> (100,250,500,750, and topspeed setting of tcpreplay). Data varies from
> malwareactivity, scanning, normal legitimat use, etc. Also used varying
> rulesets containing anything from 5000 to 25000 signatures.
> 
> What we measured:
> Looked at the stats log file to look for drops, reassembly gaps,
> fragmentation, memcaps, all of that.
> 
> Results outline:
> The higher the speed (Mbit/s) even with stats log indicating little or
> no drop, no missing reassembly, no reaced memcaps and so on, output is
> missing. With output here i mean say, alerts (unified2, fastlog, or
> EVE), and the same goes for http/dns log entries (either the old format
> or EVE).
> 
> Example #1: At 100mbit to 750 mbit we see a drastic decrase of http log
> entries (many thousand mising).
> 
> Example #2: Without pcap-log at 100Mbit/s compared to with pcap-log at
> 750Mbit/s we are missing over 80% of many log entries.
> 
> Ofcourse, there could be something i have done wrong, but with all the
> tuning, testing and experience with Suricata i cant see what else it can
> be. Has anyone else performed similar testing and or seen similar results?
> 
> Sincerly, Andreas Moe.
> 
> P.s. I know that the presented data is very vauge, but as mentiond this
> is just an initial inquery.

Since you are doing a replay, you can exactly compare the number of
packets sent vs packets received/processed by Suricata. Any significant
difference there?

Another thing to keep in mind: replay can reorder packets if NIC uses
multiple queues. See [1] for example. Reordering can mess up TCP
tracking and reassembly.

In general, replaying (TCP) traffic at faster than capture rate speeds
is problematic. I could see one case where we'd get TCP session reuse
much more often because of this. 2.1beta4 should be a lot better than
the 2.0 series in that area, but still requires ssn's to shut down more
or less cleanly.

[1] http://blog.inliniac.net/2014/02/27/tcpreplay-on-intel-82576/

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------