[Oisf-users] High packet loss with no rules
Christophe Vandeplas
christophe at vandeplas.com
Fri Jan 17 15:32:47 UTC 2014
Hi Will,
What network cards do you use?
Feel free to read my blogpost about a similar problem (and solution)
with single queue network cards
http://christophe.vandeplas.com/2013/11/suricata-capturekerneldrops-caused-by.html
On Fri, Jan 17, 2014 at 4:29 PM, Will Cladek <will.cladek at nrl.navy.mil> wrote:
> Apologies for the newbie question, but I'm experiencing a huge amount of
> packet loss on my new suricata 1.4.7 installation and can't figure out why
> or what settings I may be missing.
>
> The system has an Intel Xeon X5675 (12 cores counting hyperthreading) with
> 16 GB RAM. I routinely get 30% packet loss when running suricata on about
> 300 Mbps of traffic, even with no rules enabled. (When I just tcpdump to a
> file I see about 1% traffic loss.)
>
> The memory usage also never seem to be terribly high on the system. It'll
> be about 1% with default settings, while setting the stream max-sessions and
> prealloc-sessions to the values below brings it to around 10% without
> helping the packet loss.
>
> Is there something super-obvious I'm missing as to why I'm seeing such
> packet loss?
>
> I've included my .yaml (sans comments) at the bottom along with a sample
> run.
>
> Side question, maybe unrelated: when I set the run mode to "workers" in the
> .yaml or with the --runmode command line option, I still see in the startup
> logging:
>
> AutoFP mode using default "Active Packets" flow load balancer
>
> Does that mean it's still only using autofp, not workers?
>
> *************************************
>
> suricata.yaml:
>
> %YAML 1.1
> ---
> runmode: autofp
> default-log-dir: /var/log/suricata/
> unix-command:
> enabled: no
> outputs:
> - fast:
> enabled: yes
> filename: fast.log
> append: yes
> - unified2-alert:
> enabled: no
> filename: unified2.alert
> - http-log:
> enabled: no
> filename: http.log
> append: yes
> - tls-log:
> enabled: no # Log TLS connections.
> filename: tls.log # File to store TLS logs.
> certs-log-dir: certs # directory to store the certificates files
> - pcap-info:
> enabled: no
> - pcap-log:
> enabled: no
> filename: log.pcap
> limit: 1000mb
> max-files: 2000
> mode: normal # normal or sguil.
> use-stream-depth: no #If set to "yes" packets seen after reaching
> stream inspection depth are ignored. "no" logs all packets
> - alert-debug:
> enabled: no
> filename: alert-debug.log
> append: yes
> - alert-prelude:
> enabled: no
> profile: suricata
> log-packet-content: no
> log-packet-header: yes
> - stats:
> enabled: yes
> filename: stats.log
> interval: 8
> - syslog:
> enabled: no
> facility: local5
> - drop:
> enabled: no
> filename: drop.log
> append: yes
> - file-store:
> enabled: no # set to yes to enable
> log-dir: files # directory to store the files
> force-magic: no # force logging magic on all stored files
> force-md5: no # force logging of md5 checksums
> - file-log:
> enabled: no
> filename: files-json.log
> append: yes
> force-magic: no # force logging magic on all logged files
> force-md5: no # force logging of md5 checksums
> magic-file: /usr/share/file/magic
> nfq:
> af-packet:
> - interface: eth0
> threads: 1
> cluster-id: 99
> cluster-type: cluster_flow
> defrag: yes
> use-mmap: yes
> - interface: eth1
> threads: 16
> cluster-id: 98
> cluster-type: cluster_flow
> defrag: yes
> - interface: default
> detect-engine:
> - profile: high
> - custom-values:
> toclient-src-groups: 2
> toclient-dst-groups: 2
> toclient-sp-groups: 2
> toclient-dp-groups: 3
> toserver-src-groups: 2
> toserver-dst-groups: 4
> toserver-sp-groups: 2
> toserver-dp-groups: 25
> - sgh-mpm-context: auto
> - inspection-recursion-limit: 3000
> threading:
> set-cpu-affinity: no
> cpu-affinity:
> - management-cpu-set:
> cpu: [ 0 ] # include only these cpus in affinity settings
> - receive-cpu-set:
> cpu: [ 0 ] # include only these cpus in affinity settings
> - decode-cpu-set:
> cpu: [ 0, 1 ]
> mode: "balanced"
> - stream-cpu-set:
> cpu: [ "0-1" ]
> - detect-cpu-set:
> cpu: [ "all" ]
> mode: "exclusive" # run detect threads in these cpus
> prio:
> low: [ 0 ]
> medium: [ "1-2" ]
> high: [ 3 ]
> default: "medium"
> - verdict-cpu-set:
> cpu: [ 0 ]
> prio:
> default: "high"
> - reject-cpu-set:
> cpu: [ 0 ]
> prio:
> default: "low"
> - output-cpu-set:
> cpu: [ "all" ]
> prio:
> default: "medium"
> detect-thread-ratio: 1.5
> cuda:
> - mpm:
> packet-buffer-limit: 2400
> packet-size-limit: 1500
> packet-buffers: 10
> batching-timeout: 1
> page-locked: enabled
> device-id: 0
> cuda-streams: 2
> mpm-algo: ac
> pattern-matcher:
> - b2gc:
> search-algo: B2gSearchBNDMq
> hash-size: low
> bf-size: medium
> - b2gm:
> search-algo: B2gSearchBNDMq
> hash-size: low
> bf-size: medium
> - b2g:
> search-algo: B2gSearchBNDMq
> hash-size: low
> bf-size: medium
> - b3g:
> search-algo: B3gSearchBNDMq
> hash-size: low
> bf-size: medium
> - wumanber:
> hash-size: low
> bf-size: medium
> defrag:
> memcap: 4gb
> hash-size: 65536
> trackers: 65535 # number of defragmented flows to follow
> max-frags: 65535 # number of fragments to keep (higher than trackers)
> prealloc: yes
> timeout: 60
> flow:
> memcap: 4gb
> hash-size: 65536
> prealloc: 10000
> emergency-recovery: 30
> flow-timeouts:
> default:
> new: 30
> established: 300
> closed: 0
> emergency-new: 10
> emergency-established: 100
> emergency-closed: 0
> tcp:
> new: 60
> established: 3600
> closed: 120
> emergency-new: 10
> emergency-established: 300
> emergency-closed: 20
> udp:
> new: 30
> established: 300
> emergency-new: 10
> emergency-established: 100
> icmp:
> new: 30
> established: 300
> emergency-new: 10
> emergency-established: 100
> stream:
> memcap: 4gb
> checksum-validation: no # reject wrong csums
> max-sessions: 20000000
> prealloc-sessions: 10000000
> inline: auto # auto will use inline mode in IPS mode, yes
> or no set it statically
> reassembly:
> memcap: 8gb
> depth: 6mb # reassemble 1mb into a stream
> toserver-chunk-size: 2560
> toclient-chunk-size: 2560
> host:
> hash-size: 4096
> prealloc: 1000
> memcap: 16777216
> logging:
> default-log-level: info
> default-output-filter:
> outputs:
> - console:
> enabled: yes
> - file:
> enabled: yes
> filename: /var/log/suricata/suricata.log
> - syslog:
> enabled: no
> facility: local5
> format: "[%i] <%d> -- "
> pfring:
> - interface: eth1
> threads: 1
> cluster-id: 99
> cluster-type: cluster_flow
> - interface: default
> pcap:
> - interface: eth1
> - interface: default
> ipfw:
> default-rule-path: /etc/suricata/rules
> rule-files:
> classification-file: /etc/suricata/classification.config
> reference-config-file: /etc/suricata/reference.config
> vars:
> address-groups:
> HOME_NET: "[redacted]"
> EXTERNAL_NET: "!$HOME_NET"
> HTTP_SERVERS: "$HOME_NET"
> SMTP_SERVERS: "$HOME_NET"
> SQL_SERVERS: "$HOME_NET"
> DNS_SERVERS: "$HOME_NET"
> TELNET_SERVERS: "$HOME_NET"
> AIM_SERVERS: "$EXTERNAL_NET"
> DNP3_SERVER: "$HOME_NET"
> DNP3_CLIENT: "$HOME_NET"
> MODBUS_CLIENT: "$HOME_NET"
> MODBUS_SERVER: "$HOME_NET"
> ENIP_CLIENT: "$HOME_NET"
> ENIP_SERVER: "$HOME_NET"
> port-groups:
> HTTP_PORTS: "80"
> SHELLCODE_PORTS: "!80"
> ORACLE_PORTS: 1521
> SSH_PORTS: 22
> DNP3_PORTS: 20000
> action-order:
> - pass
> - drop
> - reject
> - alert
> host-os-policy:
> windows: [0.0.0.0/0]
> bsd: []
> bsd-right: []
> old-linux: []
> linux: [10.0.0.0/8, 192.168.1.100,
> "8762:2352:6241:7245:E000:0000:0000:0000"]
> old-solaris: []
> solaris: ["::1"]
> hpux10: []
> hpux11: []
> irix: []
> macos: []
> vista: []
> windows2k3: []
> asn1-max-frames: 256
> engine-analysis:
> rules-fast-pattern: yes
> rules: yes
> pcre:
> match-limit: 3500
> match-limit-recursion: 1500
> libhtp:
> default-config:
> personality: IDS
> request-body-limit: 3072
> response-body-limit: 3072
> request-body-minimal-inspect-size: 32kb
> request-body-inspect-window: 4kb
> response-body-minimal-inspect-size: 32kb
> response-body-inspect-window: 4kb
> double-decode-path: no
> double-decode-query: no
> server-config:
> - apache:
> address: [192.168.1.0/24, 127.0.0.0/8, "::1"]
> personality: Apache_2_2
> request-body-limit: 4096
> response-body-limit: 4096
> double-decode-path: no
> double-decode-query: no
> - iis7:
> address:
> - 192.168.0.0/24
> - 192.168.10.0/24
> personality: IIS_7_0
> request-body-limit: 4096
> response-body-limit: 4096
> double-decode-path: no
> double-decode-query: no
> profiling:
> rules:
> enabled: yes
> filename: rule_perf.log
> append: yes
> sort: avgticks
> limit: 100
> packets:
> enabled: yes
> filename: packet_stats.log
> append: yes
> csv:
> enabled: no
> filename: packet_stats.csv
> locks:
> enabled: no
> filename: lock_stats.log
> append: yes
> coredump:
> max-dump: unlimited
> napatech:
> hba: -1
> use-all-streams: yes
> streams: [1, 2, 3]
>
>
> *******************************************
>
> test run:
>
> # suricata -c /etc/suricata/suricata.yaml -i eth1
> 17/1/2014 -- 10:25:24 - <Info> - This is Suricata version 1.4.7 RELEASE
> 17/1/2014 -- 10:25:24 - <Info> - CPUs/cores online: 12
> 17/1/2014 -- 10:25:24 - <Info> - Found an MTU of 9216 for 'eth1'
> 17/1/2014 -- 10:25:24 - <Info> - allocated 3670016 bytes of memory for the
> defrag hash... 65536 buckets of size 56
> 17/1/2014 -- 10:25:24 - <Info> - preallocated 65535 defrag trackers of size
> 144
> 17/1/2014 -- 10:25:24 - <Info> - defrag memory usage: 13107056 bytes,
> maximum: 4294967296
> 17/1/2014 -- 10:25:24 - <Info> - AutoFP mode using default "Active Packets"
> flow load balancer
> 17/1/2014 -- 10:25:24 - <Info> - preallocated 1024 packets. Total memory
> 12263424
> 17/1/2014 -- 10:25:24 - <Info> - allocated 229376 bytes of memory for the
> host hash... 4096 buckets of size 56
> 17/1/2014 -- 10:25:24 - <Info> - preallocated 1000 hosts of size 120
> 17/1/2014 -- 10:25:24 - <Info> - host memory usage: 349376 bytes, maximum:
> 16777216
> 17/1/2014 -- 10:25:24 - <Info> - allocated 3670016 bytes of memory for the
> flow hash... 65536 buckets of size 56
> 17/1/2014 -- 10:25:24 - <Info> - preallocated 10000 flows of size 272
> 17/1/2014 -- 10:25:24 - <Info> - flow memory usage: 6390016 bytes, maximum:
> 4294967296
> 17/1/2014 -- 10:25:24 - <Info> - IP reputation disabled
> 17/1/2014 -- 10:25:24 - <Info> - using magic-file /usr/share/file/magic
> 17/1/2014 -- 10:25:24 - <Info> - Delayed detect disabled
> 17/1/2014 -- 10:25:24 - <Info> - No signatures supplied.
> 17/1/2014 -- 10:25:24 - <Info> - Threshold config parsed: 0 rule(s) found
> 17/1/2014 -- 10:25:24 - <Info> - Core dump size set to unlimited.
> 17/1/2014 -- 10:25:24 - <Info> - fast output device (regular) initialized:
> fast.log
> 17/1/2014 -- 10:25:24 - <Info> - Using 1 live device(s).
> 17/1/2014 -- 10:25:24 - <Info> - using interface eth1
> 17/1/2014 -- 10:25:24 - <Info> - Running in 'auto' checksum mode. Detection
> of interface state will require 1000 packets.
> 17/1/2014 -- 10:25:24 - <Info> - Found an MTU of 9216 for 'eth1'
> 17/1/2014 -- 10:25:24 - <Info> - Set snaplen to 9216 for 'eth1'
> 17/1/2014 -- 10:25:24 - <Info> - RunModeIdsPcapAutoFp initialised
> 17/1/2014 -- 10:25:24 - <Info> - stream "max-sessions": 20000000
> 17/1/2014 -- 10:25:24 - <Info> - stream "prealloc-sessions": 10000000
> 17/1/2014 -- 10:25:24 - <Info> - stream "memcap": 4294967296
> 17/1/2014 -- 10:25:24 - <Info> - stream "midstream" session pickups:
> disabled
> 17/1/2014 -- 10:25:24 - <Info> - stream "async-oneside": disabled
> 17/1/2014 -- 10:25:24 - <Info> - stream "checksum-validation": disabled
> 17/1/2014 -- 10:25:24 - <Info> - stream."inline": disabled
> 17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "memcap": 8589934592
> 17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "depth": 6291456
> 17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "toserver-chunk-size":
> 2560
> 17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "toclient-chunk-size":
> 2560
> 17/1/2014 -- 10:25:25 - <Info> - all 19 packet processing threads, 3
> management threads initialized, engine started.
> 17/1/2014 -- 10:25:25 - <Info> - No packets with invalid checksum, assuming
> checksum offloading is NOT used
> ^C17/1/2014 -- 10:26:17 - <Info> - Signal Received. Stopping engine.
> 17/1/2014 -- 10:26:17 - <Info> - 0 new flows, 0 established flows were timed
> out, 0 flows in closed state
> 17/1/2014 -- 10:26:17 - <Info> - time elapsed 53.005s
> 17/1/2014 -- 10:26:17 - <Info> - (RxPcapeth11) Packets 1373555, bytes
> 1360201182
> 17/1/2014 -- 10:26:17 - <Info> - (RxPcapeth11) Pcap Total:2188315
> Recv:1373802 Drop:814513 (37.2%).
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Total flow handler queues - 18
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 0 - pkts: 1349992
> flows: 41225
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 1 - pkts: 2829
> flows: 108
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 2 - pkts: 1351
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 3 - pkts: 1351
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 4 - pkts: 1351
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 5 - pkts: 1351
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 6 - pkts: 1351
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 7 - pkts: 1351
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 8 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 9 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 10 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 11 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 12 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 13 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 14 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 15 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 16 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 17 - pkts: 1350
> flows: 0
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 1267485 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 1300 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
> 17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
> 17/1/2014 -- 10:26:18 - <Info> - host memory usage: 349376 bytes, maximum:
> 16777216
> 17/1/2014 -- 10:26:18 - <Info> - cleaning up signature grouping structure...
> complete
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
More information about the Oisf-users
mailing list