[Oisf-users] Suricata load/latency spikes

Mon Jun 29 17:29:34 UTC 2015

My suspicion is, (and this is really just instinct) that there is a latency caused by something in the DNS side of things, and you are not going to see that in a PCAP replay unless you can stagger the replay across the duration if was collected (e.g. 90 seconds actual collection needs to be replayed synchronized to the time offset).  If this only happens a couple times a day, then you are unlikely to see it in the interval. I've had something similar before (cannot remember whether it was Suricata or Snort on dual X5650's), but it wasn't in production env so I didn't pursue it.

You wrote:

" But... a few times a day, the client network slows to an unbearable crawl. This is accompanied by a spike in the load average of the box (from around 0.2 normally, to 0.8 or above), caused by the suricata process. This lasts between 1-5 mins, then goes back to normal.

It seems the slowdown is caused by latency: pings to 8.8.8.8 go from 8ms to around 200-300ms. Pings from a non-IPSed interface are fine, so it's definitely suricata, not the rest of the network."

-----Original Message-----
From: oisf-users-bounces at lists.openinfosecfoundation.org [mailto:oisf-users-bounces at lists.openinfosecfoundation.org] On Behalf Of Oliver Humpage
Sent: Monday, June 29, 2015 1:14 PM
To: oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] Suricata load/latency spikes

Hi all,

Well I built using --enable-profiling and --enable-debug, and used almost exactly the same config as on the live router.

I had a 76MB pcap (100k packets) which had taken around 90 seconds to collect on the router. In pcap mode, suricata processed the entire thing in under a second.

The only difference is that the "workers" runmode isn't available in pcap mode, so I had to use autofp. I don't use autofp in production because when used with ipfw divert, it only has a throughput of around 130Kb. I'm not sure why.

Do you think the results above suggest that there's an issue with autofp vs workers? Or perhaps how suricata is getting the packets from ipfw? (Relevant output from pcap processing below).

Thanks again for all your help, and apologies I don't understand as much as I should about the inner workings of the suricata engine.

Oliver.

29/6/2015 -- 17:56:48 - <Info> - time elapsed 0.849s
29/6/2015 -- 17:56:48 - <Notice> - Pcap-file module read 100000 packets, 78536202 bytes
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Total flow handler queues - 6
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Queue 0  - pkts: 16570        flows: 125         
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Queue 1  - pkts: 20936        flows: 137         
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Queue 2  - pkts: 19015        flows: 243         
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Queue 3  - pkts: 16499        flows: 85          
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Queue 4  - pkts: 13556        flows: 193         
29/6/2015 -- 17:56:48 - <Info> - AutoFP - Queue 5  - pkts: 13424        flows: 227         
29/6/2015 -- 17:56:48 - <Info> - Stream TCP processed 8831 TCP packets
29/6/2015 -- 17:56:48 - <Info> - TLS logger logged 8 requests
29/6/2015 -- 17:56:48 - <Info> - HTTP logger logged 55 requests
29/6/2015 -- 17:56:48 - <Info> - DNS logger logged 0 transactions
29/6/2015 -- 17:56:48 - <Info> - Stream TCP processed 13670 TCP packets
29/6/2015 -- 17:56:48 - <Info> - TLS logger logged 0 requests
29/6/2015 -- 17:56:48 - <Info> - HTTP logger logged 25 requests
29/6/2015 -- 17:56:48 - <Info> - DNS logger logged 0 transactions
29/6/2015 -- 17:56:48 - <Info> - Stream TCP processed 11105 TCP packets
29/6/2015 -- 17:56:48 - <Info> - TLS logger logged 19 requests
29/6/2015 -- 17:56:48 - <Info> - HTTP logger logged 48 requests
29/6/2015 -- 17:56:48 - <Info> - DNS logger logged 0 transactions
29/6/2015 -- 17:56:48 - <Info> - Stream TCP processed 9217 TCP packets
29/6/2015 -- 17:56:48 - <Info> - TLS logger logged 0 requests
29/6/2015 -- 17:56:48 - <Info> - HTTP logger logged 35 requests
29/6/2015 -- 17:56:48 - <Info> - DNS logger logged 0 transactions
29/6/2015 -- 17:56:48 - <Info> - Stream TCP processed 6097 TCP packets
29/6/2015 -- 17:56:48 - <Info> - TLS logger logged 12 requests
29/6/2015 -- 17:56:48 - <Info> - HTTP logger logged 32 requests
29/6/2015 -- 17:56:48 - <Info> - DNS logger logged 0 transactions
29/6/2015 -- 17:56:48 - <Info> - Stream TCP processed 5184 TCP packets
29/6/2015 -- 17:56:48 - <Info> - TLS logger logged 4 requests
29/6/2015 -- 17:56:48 - <Info> - HTTP logger logged 34 requests
29/6/2015 -- 17:56:48 - <Info> - DNS logger logged 2 transactions
29/6/2015 -- 17:56:48 - <Info> - host memory usage: 1216000 bytes, maximum: 33554432

_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net