[Oisf-users] Updated to 2.0.6 and experiencing event logging slow down

[Victor Julien]

On 02/06/2015 06:30 AM, Gary Faulkner wrote:
> I was recently updating some sensors that I inherited from a previous
> admin from Suricata 1.4.7 to 2.0.6. The 1.4.7 install was happily
> processing around 15Gbps of traffic (multiple sensors) with 15K rules
> and less than 1% loss prior to the update. After the update I'm noticing
> that Suricata is logging fewer and fewer events as time goes on and loss
> as recorded in stats.log increases substantially over time as well. As
> an example, we record HTTP logs with Suricata and for the first few
> minutes the sensors will log thousands of events per second, but after
> 10 minutes or so that rate drops to a single event every few seconds.
> The box doesn't appear to be running out of memory, and Suricata will
> continue to run for hours in this state without crashing. Has anyone run
> into anything like this?

Can you share a portion of your stats.log?

> For some additional background I'm using Suricata with PF_RING 6.0.2 and
> the DNA drivers. PF_RING was also updated from 5.6.0 to 6.0.2. Traffic
> is load balanced across two sensors that each have 16 cores (32
> hyper-threading), 64GB RAM, and Intel 10G NICs. There were enough
> changes to the default config files between the two versions that I
> opted to diff the two and migrate our previous tuning settings to the
> new config file where appropriate, but I left the tuning settings
> largely untouched from what the previous set up was running. I've
> attached a config dump and build info if that is helpful. I suspect the
> config is suboptimal and I have experimented with some of the high
> performance tuning settings I've seen on the list, but I've largely
> found that I run out of RAM with them and Suricata never finishes
> loading. I appreciate any insight the list can offer.

Most of the 1.4 -> 2.0 changes should make things work better, not the
opposite. So I'm curious what is going on.

Couple of things to consider:

1. DNS parser was added. You don't seem to be using DNS logging so you
could disable it (set app-layer.protocols.dns.udp.enabled to no)

2. VLAN tracking was added. This is the most common source of upgrade
troubles from 1.4 to 2.0. Try with vlan.use-for-tracking set to no.

3. some of your TCP timeouts look like default. Most ppl lower them
(slash by 10) in high speed setups. Esp your flow-timeouts.tcp.closed
can be lowered.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------