[Oisf-users] High packet loss with no rules

Will Cladek will.cladek at nrl.navy.mil
Fri Jan 17 15:29:52 UTC 2014 

Apologies for the newbie question, but I'm experiencing a huge amount of packet loss on my new suricata 1.4.7 installation and can't figure out why or what settings I may be missing.

The system has an Intel Xeon X5675 (12 cores counting hyperthreading) with 16 GB RAM.  I routinely get 30% packet loss when running suricata on about 300 Mbps of traffic, even with no rules enabled.  (When I just tcpdump to a file I see about 1% traffic loss.)

The memory usage also never seem to be terribly high on the system.  It'll be about 1% with default settings, while setting the stream max-sessions and prealloc-sessions to the values below brings it to around 10% without helping the packet loss.

Is there something super-obvious I'm missing as to why I'm seeing such packet loss?

I've included my .yaml (sans comments) at the bottom along with a sample run.

Side question, maybe unrelated: when I set the run mode to "workers" in the .yaml or with the --runmode command line option, I still see in the startup logging:

AutoFP mode using default "Active Packets" flow load balancer

Does that mean it's still only using autofp, not workers?

*************************************

suricata.yaml:

%YAML 1.1
---
runmode: autofp
default-log-dir: /var/log/suricata/
unix-command:
   enabled: no
outputs:
   - fast:
       enabled: yes
       filename: fast.log
       append: yes
   - unified2-alert:
       enabled: no
       filename: unified2.alert
   - http-log:
       enabled: no
       filename: http.log
       append: yes
   - tls-log:
       enabled: no  # Log TLS connections.
       filename: tls.log # File to store TLS logs.
       certs-log-dir: certs # directory to store the certificates files
   - pcap-info:
       enabled: no
   - pcap-log:
       enabled:  no
       filename: log.pcap
       limit: 1000mb
       max-files: 2000
       mode: normal # normal or sguil.
       use-stream-depth: no #If set to "yes" packets seen after reaching stream inspection depth are ignored. "no" logs all packets
   - alert-debug:
       enabled: no
       filename: alert-debug.log
       append: yes
   - alert-prelude:
       enabled: no
       profile: suricata
       log-packet-content: no
       log-packet-header: yes
   - stats:
       enabled: yes
       filename: stats.log
       interval: 8
   - syslog:
       enabled: no
       facility: local5
   - drop:
       enabled: no
       filename: drop.log
       append: yes
   - file-store:
       enabled: no       # set to yes to enable
       log-dir: files    # directory to store the files
       force-magic: no   # force logging magic on all stored files
       force-md5: no     # force logging of md5 checksums
   - file-log:
       enabled: no
       filename: files-json.log
       append: yes
       force-magic: no   # force logging magic on all logged files
       force-md5: no     # force logging of md5 checksums
magic-file: /usr/share/file/magic
nfq:
af-packet:
   - interface: eth0
     threads: 1
     cluster-id: 99
     cluster-type: cluster_flow
     defrag: yes
     use-mmap: yes
   - interface: eth1
     threads: 16
     cluster-id: 98
     cluster-type: cluster_flow
     defrag: yes
   - interface: default
detect-engine:
   - profile: high
   - custom-values:
       toclient-src-groups: 2
       toclient-dst-groups: 2
       toclient-sp-groups: 2
       toclient-dp-groups: 3
       toserver-src-groups: 2
       toserver-dst-groups: 4
       toserver-sp-groups: 2
       toserver-dp-groups: 25
   - sgh-mpm-context: auto
   - inspection-recursion-limit: 3000
threading:
   set-cpu-affinity: no
   cpu-affinity:
     - management-cpu-set:
         cpu: [ 0 ]  # include only these cpus in affinity settings
     - receive-cpu-set:
         cpu: [ 0 ]  # include only these cpus in affinity settings
     - decode-cpu-set:
         cpu: [ 0, 1 ]
         mode: "balanced"
     - stream-cpu-set:
         cpu: [ "0-1" ]
     - detect-cpu-set:
         cpu: [ "all" ]
         mode: "exclusive" # run detect threads in these cpus
         prio:
           low: [ 0 ]
           medium: [ "1-2" ]
           high: [ 3 ]
           default: "medium"
     - verdict-cpu-set:
         cpu: [ 0 ]
         prio:
           default: "high"
     - reject-cpu-set:
         cpu: [ 0 ]
         prio:
           default: "low"
     - output-cpu-set:
         cpu: [ "all" ]
         prio:
            default: "medium"
   detect-thread-ratio: 1.5
cuda:
   - mpm:
       packet-buffer-limit: 2400
       packet-size-limit: 1500
       packet-buffers: 10
       batching-timeout: 1
       page-locked: enabled
       device-id: 0
       cuda-streams: 2
mpm-algo: ac
pattern-matcher:
   - b2gc:
       search-algo: B2gSearchBNDMq
       hash-size: low
       bf-size: medium
   - b2gm:
       search-algo: B2gSearchBNDMq
       hash-size: low
       bf-size: medium
   - b2g:
       search-algo: B2gSearchBNDMq
       hash-size: low
       bf-size: medium
   - b3g:
       search-algo: B3gSearchBNDMq
       hash-size: low
       bf-size: medium
   - wumanber:
       hash-size: low
       bf-size: medium
defrag:
   memcap: 4gb
   hash-size: 65536
   trackers: 65535 # number of defragmented flows to follow
   max-frags: 65535 # number of fragments to keep (higher than trackers)
   prealloc: yes
   timeout: 60
flow:
   memcap: 4gb
   hash-size: 65536
   prealloc: 10000
   emergency-recovery: 30
flow-timeouts:
   default:
     new: 30
     established: 300
     closed: 0
     emergency-new: 10
     emergency-established: 100
     emergency-closed: 0
   tcp:
     new: 60
     established: 3600
     closed: 120
     emergency-new: 10
     emergency-established: 300
     emergency-closed: 20
   udp:
     new: 30
     established: 300
     emergency-new: 10
     emergency-established: 100
   icmp:
     new: 30
     established: 300
     emergency-new: 10
     emergency-established: 100
stream:
   memcap: 4gb
   checksum-validation: no      # reject wrong csums
   max-sessions: 20000000
   prealloc-sessions: 10000000
   inline: auto                  # auto will use inline mode in IPS mode, yes or no set it statically
   reassembly:
     memcap: 8gb
     depth: 6mb                  # reassemble 1mb into a stream
     toserver-chunk-size: 2560
     toclient-chunk-size: 2560
host:
   hash-size: 4096
   prealloc: 1000
   memcap: 16777216
logging:
   default-log-level: info
   default-output-filter:
   outputs:
   - console:
       enabled: yes
   - file:
       enabled: yes
       filename: /var/log/suricata/suricata.log
   - syslog:
       enabled: no
       facility: local5
       format: "[%i] <%d> -- "
pfring:
   - interface: eth1
     threads: 1
     cluster-id: 99
     cluster-type: cluster_flow
   - interface: default
pcap:
   - interface: eth1
   - interface: default
ipfw:
default-rule-path: /etc/suricata/rules
rule-files:
classification-file: /etc/suricata/classification.config
reference-config-file: /etc/suricata/reference.config
vars:
   address-groups:
     HOME_NET: "[redacted]"
     EXTERNAL_NET: "!$HOME_NET"
     HTTP_SERVERS: "$HOME_NET"
     SMTP_SERVERS: "$HOME_NET"
     SQL_SERVERS: "$HOME_NET"
     DNS_SERVERS: "$HOME_NET"
     TELNET_SERVERS: "$HOME_NET"
     AIM_SERVERS: "$EXTERNAL_NET"
     DNP3_SERVER: "$HOME_NET"
     DNP3_CLIENT: "$HOME_NET"
     MODBUS_CLIENT: "$HOME_NET"
     MODBUS_SERVER: "$HOME_NET"
     ENIP_CLIENT: "$HOME_NET"
     ENIP_SERVER: "$HOME_NET"
   port-groups:
     HTTP_PORTS: "80"
     SHELLCODE_PORTS: "!80"
     ORACLE_PORTS: 1521
     SSH_PORTS: 22
     DNP3_PORTS: 20000
action-order:
   - pass
   - drop
   - reject
   - alert
host-os-policy:
   windows: [0.0.0.0/0]
   bsd: []
   bsd-right: []
   old-linux: []
   linux: [10.0.0.0/8, 192.168.1.100, "8762:2352:6241:7245:E000:0000:0000:0000"]
   old-solaris: []
   solaris: ["::1"]
   hpux10: []
   hpux11: []
   irix: []
   macos: []
   vista: []
   windows2k3: []
asn1-max-frames: 256
engine-analysis:
   rules-fast-pattern: yes
   rules: yes
pcre:
   match-limit: 3500
   match-limit-recursion: 1500
libhtp:
    default-config:
      personality: IDS
      request-body-limit: 3072
      response-body-limit: 3072
      request-body-minimal-inspect-size: 32kb
      request-body-inspect-window: 4kb
      response-body-minimal-inspect-size: 32kb
      response-body-inspect-window: 4kb
      double-decode-path: no
      double-decode-query: no
    server-config:
      - apache:
          address: [192.168.1.0/24, 127.0.0.0/8, "::1"]
          personality: Apache_2_2
          request-body-limit: 4096
          response-body-limit: 4096
          double-decode-path: no
          double-decode-query: no
      - iis7:
          address:
            - 192.168.0.0/24
            - 192.168.10.0/24
          personality: IIS_7_0
          request-body-limit: 4096
          response-body-limit: 4096
          double-decode-path: no
          double-decode-query: no
profiling:
   rules:
     enabled: yes
     filename: rule_perf.log
     append: yes
     sort: avgticks
     limit: 100
   packets:
     enabled: yes
     filename: packet_stats.log
     append: yes
     csv:
       enabled: no
       filename: packet_stats.csv
   locks:
     enabled: no
     filename: lock_stats.log
     append: yes
coredump:
   max-dump: unlimited
napatech:
     hba: -1
     use-all-streams: yes
     streams: [1, 2, 3]


*******************************************

test run:

# suricata -c /etc/suricata/suricata.yaml -i eth1
17/1/2014 -- 10:25:24 - <Info> - This is Suricata version 1.4.7 RELEASE
17/1/2014 -- 10:25:24 - <Info> - CPUs/cores online: 12
17/1/2014 -- 10:25:24 - <Info> - Found an MTU of 9216 for 'eth1'
17/1/2014 -- 10:25:24 - <Info> - allocated 3670016 bytes of memory for the defrag hash... 65536 buckets of size 56
17/1/2014 -- 10:25:24 - <Info> - preallocated 65535 defrag trackers of size 144
17/1/2014 -- 10:25:24 - <Info> - defrag memory usage: 13107056 bytes, maximum: 4294967296
17/1/2014 -- 10:25:24 - <Info> - AutoFP mode using default "Active Packets" flow load balancer
17/1/2014 -- 10:25:24 - <Info> - preallocated 1024 packets. Total memory 12263424
17/1/2014 -- 10:25:24 - <Info> - allocated 229376 bytes of memory for the host hash... 4096 buckets of size 56
17/1/2014 -- 10:25:24 - <Info> - preallocated 1000 hosts of size 120
17/1/2014 -- 10:25:24 - <Info> - host memory usage: 349376 bytes, maximum: 16777216
17/1/2014 -- 10:25:24 - <Info> - allocated 3670016 bytes of memory for the flow hash... 65536 buckets of size 56
17/1/2014 -- 10:25:24 - <Info> - preallocated 10000 flows of size 272
17/1/2014 -- 10:25:24 - <Info> - flow memory usage: 6390016 bytes, maximum: 4294967296
17/1/2014 -- 10:25:24 - <Info> - IP reputation disabled
17/1/2014 -- 10:25:24 - <Info> - using magic-file /usr/share/file/magic
17/1/2014 -- 10:25:24 - <Info> - Delayed detect disabled
17/1/2014 -- 10:25:24 - <Info> - No signatures supplied.
17/1/2014 -- 10:25:24 - <Info> - Threshold config parsed: 0 rule(s) found
17/1/2014 -- 10:25:24 - <Info> - Core dump size set to unlimited.
17/1/2014 -- 10:25:24 - <Info> - fast output device (regular) initialized: fast.log
17/1/2014 -- 10:25:24 - <Info> - Using 1 live device(s).
17/1/2014 -- 10:25:24 - <Info> - using interface eth1
17/1/2014 -- 10:25:24 - <Info> - Running in 'auto' checksum mode. Detection of interface state will require 1000 packets.
17/1/2014 -- 10:25:24 - <Info> - Found an MTU of 9216 for 'eth1'
17/1/2014 -- 10:25:24 - <Info> - Set snaplen to 9216 for 'eth1'
17/1/2014 -- 10:25:24 - <Info> - RunModeIdsPcapAutoFp initialised
17/1/2014 -- 10:25:24 - <Info> - stream "max-sessions": 20000000
17/1/2014 -- 10:25:24 - <Info> - stream "prealloc-sessions": 10000000
17/1/2014 -- 10:25:24 - <Info> - stream "memcap": 4294967296
17/1/2014 -- 10:25:24 - <Info> - stream "midstream" session pickups: disabled
17/1/2014 -- 10:25:24 - <Info> - stream "async-oneside": disabled
17/1/2014 -- 10:25:24 - <Info> - stream "checksum-validation": disabled
17/1/2014 -- 10:25:24 - <Info> - stream."inline": disabled
17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "memcap": 8589934592
17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "depth": 6291456
17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "toserver-chunk-size": 2560
17/1/2014 -- 10:25:24 - <Info> - stream.reassembly "toclient-chunk-size": 2560
17/1/2014 -- 10:25:25 - <Info> - all 19 packet processing threads, 3 management threads initialized, engine started.
17/1/2014 -- 10:25:25 - <Info> - No packets with invalid checksum, assuming checksum offloading is NOT used
^C17/1/2014 -- 10:26:17 - <Info> - Signal Received.  Stopping engine.
17/1/2014 -- 10:26:17 - <Info> - 0 new flows, 0 established flows were timed out, 0 flows in closed state
17/1/2014 -- 10:26:17 - <Info> - time elapsed 53.005s
17/1/2014 -- 10:26:17 - <Info> - (RxPcapeth11) Packets 1373555, bytes 1360201182
17/1/2014 -- 10:26:17 - <Info> - (RxPcapeth11) Pcap Total:2188315 Recv:1373802 Drop:814513 (37.2%).
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Total flow handler queues - 18
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 0  - pkts: 1349992      flows: 41225
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 1  - pkts: 2829         flows: 108
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 2  - pkts: 1351         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 3  - pkts: 1351         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 4  - pkts: 1351         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 5  - pkts: 1351         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 6  - pkts: 1351         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 7  - pkts: 1351         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 8  - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 9  - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 10 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 11 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 12 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 13 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 14 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 15 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 16 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - AutoFP - Queue 17 - pkts: 1350         flows: 0
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 1267485 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 1300 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:17 - <Info> - Stream TCP processed 0 TCP packets
17/1/2014 -- 10:26:17 - <Info> - Fast log output wrote 0 alerts
17/1/2014 -- 10:26:18 - <Info> - host memory usage: 349376 bytes, maximum: 16777216
17/1/2014 -- 10:26:18 - <Info> - cleaning up signature grouping structure... complete

More information about the Oisf-users mailing list