[Oisf-users] Help with 99% CPU usage
Duarte Silva
duarte.silva at serializing.me
Wed May 15 10:25:32 UTC 2013
Hi all,
I'm currently facing a problem with Suricata. After running for a while, there
is always an AF_PACKET thread (workers mode) that hogs the CPU to which it is
bound to creating an awful amount of packet loss. I have discarded the
following factors:
- Number of rules, it has also happened without rules;
- Amount of network traffic, I have seen Suricata handle ~18 MBs (150 MBps) of
traffic without problems with the current configuration and it as also happened
with only ~2 MBs of traffic;
- Memory, Suricata was only using ~500 MB of it when the CPU usage pegged to
100%;
This happens repeatedly and after it happens, Suricata takes a long time to
stop. Could some tell me what I can do to debug this issue?
Suricata is executed with the following command line:
suricata -D -c /etc/suricata/suricata.yaml --pidfile /var/lock/subsys/suricata
--af-packet=eth1 --user=suri --group=suri
I have also attached any files that can help out in debugging.
Thanks in advance,
Duarte Silva
-------------- next part --------------
This is Suricata version 1.4.1 RELEASE
Features: PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK HAVE_HTP_TX_GET_RESPONSE_HEADERS_RAW HAVE_NSS HAVE_LIBJANSSON
64-bits, Little-endian architecture
GCC version 4.4.7 20120313 (Red Hat 4.4.7-3), C version 199901
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_2
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_4
__GCC_HAVE_SYNC_COMPARE_AND_SWAP_8
compiled with -fstack-protector-all
compiled with _FORTIFY_SOURCE=2
compiled with libhtp 0.2.12, linked against 0.2.12
Suricata Configuration:
AF_PACKET support: yes
PF_RING support: no
NFQueue support: no
IPFW support: no
DAG enabled: no
Napatech enabled: no
Unix socket enabled: yes
libnss support: yes
libnspr support: yes
libjansson support: yes
Prelude support: no
PCRE jit: no
libluajit: no
libgeoip: yes
Non-bundled htp: no
Old barnyard2 support: no
CUDA enabled: no
Suricatasc install: yes
Unit tests enabled: no
Debug output enabled: no
Debug validation enabled: no
Profiling enabled: no
Profiling locks enabled: no
Generic build parameters:
Installation prefix (--prefix): /usr
Configuration directory (--sysconfdir): /etc/suricata/
Log directory (--localstatedir) : /var/log/suricata/
Host: x86_64-redhat-linux-gnu
GCC binary: gcc -std=gnu99
GCC Protect enabled: yes
GCC march native enabled: no
GCC Profile enabled: no
-------------- next part --------------
14/5/2013 -- 22:23:16 - <Info> - allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
14/5/2013 -- 22:23:16 - <Info> - preallocated 65535 defrag trackers of size 144
14/5/2013 -- 22:23:16 - <Info> - defrag memory usage: 13107056 bytes, maximum: 33554432
14/5/2013 -- 22:23:16 - <Info> - AutoFP mode using default "Active Packets" flow load balancer
14/5/2013 -- 22:23:16 - <Info> - preallocated 1024 packets. Total memory 4362240
14/5/2013 -- 22:23:16 - <Info> - allocated 229376 bytes of memory for the host hash... 4096 buckets of size 56
14/5/2013 -- 22:23:16 - <Info> - preallocated 1000 hosts of size 120
14/5/2013 -- 22:23:16 - <Info> - host memory usage: 349376 bytes, maximum: 16777216
14/5/2013 -- 22:23:16 - <Info> - allocated 3670016 bytes of memory for the flow hash... 65536 buckets of size 56
14/5/2013 -- 22:23:16 - <Info> - preallocated 10000 flows of size 272
14/5/2013 -- 22:23:16 - <Info> - flow memory usage: 6390016 bytes, maximum: 33554432
14/5/2013 -- 22:23:16 - <Info> - IP reputation disabled
14/5/2013 -- 22:23:16 - <Info> - using magic-file /usr/share/file/magic
14/5/2013 -- 22:23:16 - <Info> - Delayed detect disabled
14/5/2013 -- 22:23:17 - <Info> - 1 rule files processed. 1215 rules successfully loaded, 0 rules failed
14/5/2013 -- 22:23:17 - <Info> - 1215 signatures processed. 1 are IP-only rules, 573 are inspecting packet payload, 1015 inspect application layer, 0 are decoder event only
14/5/2013 -- 22:23:17 - <Info> - building signature grouping structure, stage 1: adding signatures to signature source addresses... complete
14/5/2013 -- 22:23:17 - <Info> - building signature grouping structure, stage 2: building source address list... complete
14/5/2013 -- 22:23:17 - <Info> - building signature grouping structure, stage 3: building destination address lists... complete
14/5/2013 -- 22:23:17 - <Info> - Threshold config parsed: 0 rule(s) found
14/5/2013 -- 22:23:17 - <Info> - Core dump size set to unlimited.
14/5/2013 -- 22:23:17 - <Info> - dropped the caps for main thread
14/5/2013 -- 22:23:17 - <Info> - fast output device (regular) initialized: fast.log
14/5/2013 -- 22:23:17 - <Info> - Unified2-alert initialized: filename unified2.alert, limit 32 MB
14/5/2013 -- 22:23:17 - <Info> - http-log output device (regular) initialized: http.log
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "management-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Using default prio 'low'
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "receive-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "decode-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "stream-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "detect-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Using default prio 'high'
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "verdict-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Using default prio 'high'
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "reject-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Using default prio 'low'
14/5/2013 -- 22:23:17 - <Info> - Found affinity definition for "output-cpu-set"
14/5/2013 -- 22:23:17 - <Info> - Using default prio 'medium'
14/5/2013 -- 22:23:17 - <Info> - Enabling mmaped capture on iface eth1
14/5/2013 -- 22:23:17 - <Info> - Using flow cluster mode for AF_PACKET (iface eth1)
14/5/2013 -- 22:23:17 - <Info> - Using defrag kernel functionality for AF_PACKET (iface eth1)
14/5/2013 -- 22:23:17 - <Info> - Going to use bpf filter host <scrubbed>
14/5/2013 -- 22:23:17 - <Info> - Going to use 2 thread(s)
14/5/2013 -- 22:23:17 - <Info> - Setting affinity on CPU 0
14/5/2013 -- 22:23:17 - <Info> - Setting prio 2 for "AFPacketeth11" Module to cpu/core 0, thread id 6245
14/5/2013 -- 22:23:17 - <Info> - Enabling zero copy mode
14/5/2013 -- 22:23:17 - <Info> - Enabling zero copy mode by using data release call
14/5/2013 -- 22:23:17 - <Info> - Setting affinity on CPU 1
14/5/2013 -- 22:23:17 - <Info> - Setting prio 0 for "AFPacketeth12" Module to cpu/core 1, thread id 6246
14/5/2013 -- 22:23:17 - <Info> - Enabling zero copy mode
14/5/2013 -- 22:23:17 - <Info> - Enabling zero copy mode by using data release call
14/5/2013 -- 22:23:17 - <Info> - RunModeIdsAFPWorkers initialised
14/5/2013 -- 22:23:17 - <Info> - Setting prio 2 for "FlowManagerThread" thread , thread id 6247
14/5/2013 -- 22:23:17 - <Info> - stream "max-sessions": 262144
14/5/2013 -- 22:23:17 - <Info> - stream "prealloc-sessions": 32768
14/5/2013 -- 22:23:17 - <Info> - stream "memcap": 805306368
14/5/2013 -- 22:23:17 - <Info> - stream "midstream" session pickups: disabled
14/5/2013 -- 22:23:17 - <Info> - stream "async-oneside": disabled
14/5/2013 -- 22:23:17 - <Info> - stream "checksum-validation": enabled
14/5/2013 -- 22:23:17 - <Info> - stream."inline": disabled
14/5/2013 -- 22:23:17 - <Info> - stream.reassembly "memcap": 939524096
14/5/2013 -- 22:23:17 - <Info> - stream.reassembly "depth": 8388608
14/5/2013 -- 22:23:17 - <Info> - stream.reassembly "toserver-chunk-size": 2560
14/5/2013 -- 22:23:17 - <Info> - stream.reassembly "toclient-chunk-size": 2560
14/5/2013 -- 22:23:17 - <Info> - Setting prio 2 for "SCPerfWakeupThread" thread , thread id 6248
14/5/2013 -- 22:23:17 - <Info> - Setting prio 2 for "SCPerfMgmtThread" thread , thread id 6249
14/5/2013 -- 22:23:17 - <Info> - all 2 packet processing threads, 3 management threads initialized, engine started.
14/5/2013 -- 22:23:17 - <Info> - AF_PACKET RX Ring params: block_size=32768 block_nr=1876 frame_size=1584 frame_nr=37520
14/5/2013 -- 22:23:17 - <Info> - Using interface 'eth1' via socket 8
14/5/2013 -- 22:23:17 - <Info> - Using BPF 'host <scrubbed>' on iface 'eth1'
14/5/2013 -- 22:23:17 - <Info> - Thread AFPacketeth11 using socket 8
14/5/2013 -- 22:23:17 - <Info> - AF_PACKET RX Ring params: block_size=32768 block_nr=1876 frame_size=1584 frame_nr=37520
14/5/2013 -- 22:23:17 - <Info> - Using interface 'eth1' via socket 9
14/5/2013 -- 22:23:17 - <Info> - Using BPF 'host <scrubbed>' on iface 'eth1'
14/5/2013 -- 22:23:17 - <Info> - All AFP capture threads are running.
14/5/2013 -- 22:23:17 - <Info> - Thread AFPacketeth12 using socket 9
15/5/2013 -- 10:48:13 - <Info> - Signal Received. Stopping engine.
15/5/2013 -- 10:48:13 - <Info> - 0 new flows, 0 established flows were timed out, 0 flows in closed state
15/5/2013 -- 10:48:13 - <Info> - time elapsed 44696.762s
15/5/2013 -- 10:48:13 - <Info> - (AFPacketeth11) Kernel: Packets 28082083, dropped 0
15/5/2013 -- 10:48:13 - <Info> - (AFPacketeth11) Packets 28081845, bytes 11039726314
15/5/2013 -- 10:48:13 - <Info> - Stream TCP processed 25285195 TCP packets
15/5/2013 -- 10:48:13 - <Info> - Fast log output wrote 0 alerts
15/5/2013 -- 10:48:13 - <Info> - Alert unified2 module wrote 0 alerts
15/5/2013 -- 10:48:13 - <Info> - HTTP logger logged 135249 requests
15/5/2013 -- 10:48:13 - <Info> - (AFPacketeth12) Kernel: Packets 28380393, dropped 68029
15/5/2013 -- 10:48:13 - <Info> - (AFPacketeth12) Packets 28289525, bytes 11639366285
15/5/2013 -- 10:48:13 - <Info> - Stream TCP processed 25343628 TCP packets
15/5/2013 -- 10:48:13 - <Info> - Fast log output wrote 0 alerts
15/5/2013 -- 10:48:13 - <Info> - HTTP logger logged 125905 requests
15/5/2013 -- 10:51:37 - <Info> - host memory usage: 349376 bytes, maximum: 16777216
15/5/2013 -- 10:51:37 - <Info> - cleaning up signature grouping structure... complete
-------------- next part --------------
A non-text attachment was scrubbed...
Name: suricata.yaml
Type: application/x-yaml
Size: 39018 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20130515/e6788e2d/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: stats.tar.gz
Type: application/x-compressed-tar
Size: 735852 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20130515/e6788e2d/attachment-0003.bin>
More information about the Oisf-users
mailing list