[Oisf-users] Suricata Myricom and 10Gbit

[Victor Julien]

On 03/31/2014 01:27 PM, Michał Purzyński wrote:
> Hello.
> 
> I'm trying to tune Suricata to handle up to 10Gbit/sec of traffic
> (that's a peak, jumps like crazy from 2.5 - 4.5 - 6 and up). So far my
> results were quite bad, so I'm seeking help - must be missing something
> obvious here judging by the numbers of articles where everyone seems to
> use Suricata on 10Gbit traffic.
> 
> Server:
> 
> 2 x Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz (16 physical cores)
> 64GB RAM
> 
> NIC - Myricom 10Gb 10G-PCIE-8B-S with the Sniffer software loaded and
> activated
> 
> Software:
> 
> This is Suricata version 2.0rc2 RELEASE
> 
> Command line:
> 
> SNF_NUM_RINGS=16 SNF_FLAGS=0x1 SNF_DESCRING_SIZE=1073741824
> SNF_DATARING_SIZE=1073741824 SNF_DEBUG_MASK=0x3 suricata -c
> /etc/nsm/nsm11-eth4/suricata.yaml -i eth4 --runmode=workers
> 
> (16 threads, 1GB for each buffer)
> 
> The Myricom debug output seems fine.
> 
> Config file - pretty standard, most important things:
> 
> max-pending-packets: 5000
> runmode: workers
> 
> detect-engine:
>   - profile: medium
> 
> Did not touch parameters here.
> 
>   set-cpu-affinity: no
> 
> Also default settings here.
> 
>   detect-thread-ratio: 1.5
> 
> (should not it be 1.0?)

It's not used in runmode workers

> 
> defrag:
>   memcap: 512mb
>   trackers: 65535 # number of defragmented flows to follow
>   max-frags: 65535 # number of fragments to keep (higher than trackers)
>   prealloc: yes
>   timeout: 60
> 
> flow:
>   memcap: 32mb
>   hash-size: 65536
>   prealloc: 10000

Definitely increase all these settings. In our 10g setup we use:

flow:
  memcap: 3200mb
  hash-size: 15728640
  prealloc: 8000000


>   emergency-recovery: 30
> 
> stream:
>   memcap: 16gb
>   max-sessions: 20000000
>   prealloc-sessions: 10000000
>   checksum-validation: yes      # reject wrong csums
>   inline: no                    # no inline mode
>   reassembly:
>     memcap: 14gb
>     depth: 6mb                  # reassemble 1mb into a stream
>     toserver-chunk-size: 2560
>     toclient-chunk-size: 2560
> 
> pcap:
>   - interface: eth4
>     threads: 16
>     buffer-size: 512kb

Not sure how this buffer size relates to myricom's libpcap. Perhaps you
can try to increase it.

>     checksum-checks: no
> 
> The myricom tools show a high packet loss
> 
>                      SNF recv pkts:            634485790
>                 SNF drop ring full:            137774061
>                         Interrupts:             12053363
>            Net bad PHY/CRC32 drop:                32092
>                  Net overflow drop:               219656
> 
> Also note that it reports quite a few interrupts, which there should be
> almost none.
> 
> What is the direction I should go here? I know that tuning a high
> capacity Suricata isn't exactly a single afternoon task, but I need to
> advise what to do now, how to proceed, etc.
> 
> Looking for clues.

How is the cpu use looking? All cores busy?

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------