[Oisf-users] Suricata pegs a detect thread and drops packets

[David Vasil]

I have been trying to track down an issue I am having with Suricata
dropping packets (seems to be a theme on this list), requiring a restart of
the daemon to clear the condition.  My environment is not large (averge
40-80Mbps traffic, mostly user/http traffic) and I have Suricata 2.0.1
running on a base installation of Security Onion 12.04.4 on a Dell R610
(12GB RAM, Dual Intel X5570, Broadcom BCM5709 sniffing interface).

About once a day, Zabbix shows that I am starting to see a large number of
capture.kernel_drops and some corresponding tcp.reassembly_gap.  Looking at
htop, I can see that one of the Detect threads (Detect1 in this screenshot)
is pegged at 100% utilization.  If I use 'perf top' to look at the perf
events on the system, I see libhtp consuming a large number of the cycles
(attached).  Restarting suricata using 'nsm_sensor_stop --only-snort-alert'
results in child threads exiting, but the main suricata process itself
never stops (requiring a kill -9).  Starting suricata again with
'nsm_sensor_start --only-snort-alert' starts up Suricata and shows that we
are able to inspect traffic with no drops.

In the attached screenshots, I am only inspecting ~2k packets/sec ~16Mbit/s
when Suricata started dropping packets.  As I write this, Suricata is
processing ~7k packets/sec and ~40Mbit/s with no drops.  I could not see
anything that I can directly correlate to the drops and the various tuning
steps I have taken have not helped alleviate the issue, so I was hoping to
leverage the community's wisdom.

Some observations I had:

- Bro (running on the same system, on the same interface) drops 0% packets
without issue all day
- When I start seeing capture.kernel_drops, I also begin seeing an uptick
in flow_mgr.new_pruned and tcp.reassembly_gap, changing the associated
memcaps of each has not seemed to help
- tcp.reassembly_memuse jumps to a peak of around 2.66G even though my
reassembly memcap is set to 2gb
- http.memcap is set to 256mb in my config and logfile, but the stats.log
show http.memcap = 0 (bug?)

Thanks!
-dave

suricata.yaml: http://pastebin.com/b73wC2B3
suricata.log (not from the same instance of drops as the screenshots, but
during a period of similar symptoms): http://pastebin.com/yyrxWxT9
stats.log (not from the same instance of drops as the screenshots, but
during a period of similar symptoms): http://pastebin.com/z6GF5sH2
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140618/a8b987c5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: suricata_detect1-pegged0.png
Type: image/png
Size: 184751 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140618/a8b987c5/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zabbix0.png
Type: image/png
Size: 705735 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140618/a8b987c5/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: htp_list_array_get-pegged0.png
Type: image/png
Size: 171337 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140618/a8b987c5/attachment-0005.png>