[Oisf-users] Suricata Myricom and 10Gbit

[Erich Lerch]

Michał,

We have a similar setup, also with the Myricom 10gb interface.

The following values give us fairly good results, although our traffic
does not exceed 2Gbit at the moment.
Performance also depends on the ruleset.
If you have profiling compiled in, try without.

SNF settings:
SNF_NUM_RINGS=16  SNF_DATARING_SIZE=34359738368 SNF_DESCRING_SIZE=2147483648

Suricata settings:

...
max-pending-packets: 2048
...
# very important:
detect-engine:
  - profile: custom
  - custom-values:
      toclient-src-groups: 200
      toclient-dst-groups: 200
      toclient-sp-groups: 200
      toclient-dp-groups: 300
      toserver-src-groups: 200
      toserver-dst-groups: 400
      toserver-sp-groups: 200
      toserver-dp-groups: 200
  - sgh-mpm-context: single
  - inspection-recursion-limit: 3000
  - rule-reload: true
...
threading:
  set-cpu-affinity: yes
  [definition of cpu sets]
...
defrag:
  memcap: 512mb
  hash-size: 65536
  trackers: 65535
  max-frags: 65535
  prealloc: yes
  timeout: 10
...
flow:
  memcap: 256mb
  hash-size: 262144
  prealloc: 300000
  emergency-recovery: 30
...
flow-timeouts:
  [a lot more aggressive than the default!!!]
...
stream:
  memcap: 12gb
  checksum-validation: no
  inline: no
  prealloc-sessions: 10000000
  reassembly:
    memcap: 16gb
    depth: 6mb                 # reassemble 1mb into a stream
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560
    randomize-chunk-size: yes
...

pcap:
  - interface: snf0
    threads: 16
    buffer-size: 512mb
    checksum-checks: no
    promisc: no
...

Also set the app-layer values much higher than the defaults.

cheers,
erich


2014-03-31 13:27 GMT+02:00 Michał Purzyński <michalpurzynski1 at gmail.com>:
> Hello.
>
> I'm trying to tune Suricata to handle up to 10Gbit/sec of traffic (that's a
> peak, jumps like crazy from 2.5 - 4.5 - 6 and up). So far my results were
> quite bad, so I'm seeking help - must be missing something obvious here
> judging by the numbers of articles where everyone seems to use Suricata on
> 10Gbit traffic.
>
> Server:
>
> 2 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (16 physical cores)
> 64GB RAM
>
> NIC - Myricom 10Gb 10G-PCIE-8B-S with the Sniffer software loaded and
> activated
>
> Software:
>
> This is Suricata version 2.0rc2 RELEASE
>
> Command line:
>
> SNF_NUM_RINGS=16 SNF_FLAGS=0x1 SNF_DESCRING_SIZE=1073741824
> SNF_DATARING_SIZE=1073741824 SNF_DEBUG_MASK=0x3 suricata -c
> /etc/nsm/nsm11-eth4/suricata.yaml -i eth4 --runmode=workers
>
> (16 threads, 1GB for each buffer)
>
> The Myricom debug output seems fine.
>
> Config file - pretty standard, most important things:
>
> max-pending-packets: 5000
> runmode: workers
>
> detect-engine:
>   - profile: medium
>
> Did not touch parameters here.
>
>   set-cpu-affinity: no
>
> Also default settings here.
>
>   detect-thread-ratio: 1.5
>
> (should not it be 1.0?)
>
> defrag:
>   memcap: 512mb
>   trackers: 65535 # number of defragmented flows to follow
>   max-frags: 65535 # number of fragments to keep (higher than trackers)
>   prealloc: yes
>   timeout: 60
>
> flow:
>   memcap: 32mb
>   hash-size: 65536
>   prealloc: 10000
>   emergency-recovery: 30
>
> stream:
>   memcap: 16gb
>   max-sessions: 20000000
>   prealloc-sessions: 10000000
>   checksum-validation: yes      # reject wrong csums
>   inline: no                    # no inline mode
>   reassembly:
>     memcap: 14gb
>     depth: 6mb                  # reassemble 1mb into a stream
>     toserver-chunk-size: 2560
>     toclient-chunk-size: 2560
>
> pcap:
>   - interface: eth4
>     threads: 16
>     buffer-size: 512kb
>     checksum-checks: no
>
> The myricom tools show a high packet loss
>
>                      SNF recv pkts:            634485790
>                 SNF drop ring full:            137774061
>                         Interrupts:             12053363
>            Net bad PHY/CRC32 drop:                32092
>                  Net overflow drop:               219656
>
> Also note that it reports quite a few interrupts, which there should be
> almost none.
>
> What is the direction I should go here? I know that tuning a high capacity
> Suricata isn't exactly a single afternoon task, but I need to advise what to
> do now, how to proceed, etc.
>
> Looking for clues.
>
> --
> Michał Purzyński
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/