[Oisf-users] Fwd: Installing / Running Suricata with Myricom NICs

[Erich Lerch]

Hi Alex

We seem to have a similar setup as of OS, hardware, traffic and myricom.
We experience almost no drops (usually less than 0.2%).

We start suri with these params:

SNF_NUM_RINGS=10 SNF_FLAGS=0x1 SNF_DATARING_SIZE=12884901888
SNF_DESCRING_SIZE=3221225472

We do start Suri with "-i snf0", it throws a warning in suricata.log, but
it works:
<Warning> - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get
feature via ioctl for 'snf0': No such device (19)


There are other factors which influence the performance:

- I set a BPF to bypass traffic I don't want to see anyway

- number of rules (we have activated about 22'000 rules)

- suricata.yaml, some configs greatly influence performance
Some excerpts:
...
pcap:
  - interface: snf0
    threads: 10   <<--- has to correspond with SNF_NUM_RINGS
    buffer-size: 2gb
    checksum-checks: no
    promisc: no
    snaplen: 1520
    bpf-filter: "..."

...
stream:
  memcap: 5gb
...
    reassembly:
    memcap: 10gb
...

detect:
  profile: custom
  custom-values:
    toclient-groups: 200
    toserver-groups: 200
...
# hyperscan
mpm-algo: hs
spm-algo: hs

# pin to cores
threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ ... ]
    - worker-cpu-set:
        cpu: [ ... ]
        mode: "exclusive" # run detect threads in these cpus
        threads: 10
        prio:
          high: [ ... ]
          default: "medium"
...
max-pending-packets: 8192
...
flow:
  memcap: 1024mb
  hash-size: 524288
  prealloc: 1048576
  emergency-recovery: 30
  managers: 2
  recyclers: 2
...


- try to pin suri worker threads to the same NUMA node the myricom is
attached to


HTH,
erich


2018-02-20 18:58 GMT+01:00 Alexander Merck <alexander.merck at duke.edu>:

> Hello,
>
>
>
> Hopefully someone can help shed some light on some issues we've been
> seeing. We just installed a new instance of Suricata on a fresh RHEL7
> monitoring box with Myricom cards. However, we are seeing significant
> packet loss (20-35%) on 2-3 Gbps traffic when attempting to use the SNF
> drivers.
>
>
>
>
>
> I'm suspecting that the Myricom SNF drivers are not functioning as
> expected. We're able to run tcpdump compiled against these drivers with no
> issue, including generating debug output. We've also found when supplying
> the SNF_DEBUG_MASK environment variable when running Suricata, no debug
> output is generated.
>
>
>
> Also, when using Suricata with the SNF drivers, should you be able to use
> the interface names specified by SNF (e.g. snf0)? When trying to run
> Suricata using the -i snf0, we get an "Unable to find iface snf0: No such
> device" error message. We are only able to run Suricata against the
> interface names specified by the kernel (in our case, enp4s0)
>
>
>
>
>
> The version of Suricata is 4.0.4 and the version of SNF is 3.0.12. Running
> ldd shows that Suricata is linked against the SNF libraries.
>
>
>
> # ldd /usr/bin/suricata
>
> ...
>
>         libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007f238ffb0000)
>
> ...
>
>         libsnf.so.0 => /opt/snf/lib/libsnf.so.0 (0x00007f238dae4000)
>
>
>
>
>
> We compiled Suricata per these instructions:
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom.
> I did notice that this document is over five years old, but all of the
> configuration options seemed correct.
>
>
>
> ./configure --with-libpcap-includes=/opt/snf/include/
> --with-libpcap-libraries=/opt/snf/lib/ --prefix=/usr --sysconfdir=/etc
> --localstatedir=/var
>
>
>
> And we're running Suricata with the following command:
>
>
>
> SNF_NUM_RINGS=32 SNF_DATARING_SIZE=17179869184
> SNF_DESCRING_SIZE=4294967296 SNF_FLAGS=0x1 SNF_DEBUG_MASK=3
> SNF_DEBUG_FILENAME="/tmp/snf.out" /usr/bin/suricata -c
> /etc/suricata/suricata.yaml -i enp4s0 --runmode=workers
>
>
>
> The box we’re running this on has 64 cores and 256GB of RAM, so I doubt
> it’s a resource issue…but could potentially be a configuration issue.
>
>
>
>
>
> Are we missing something in the install process that may be causing these
> issues? Any recommendations or pointers would be greatly appreciated.
> Thanks!
>
>
>
> -Alex M
>
>
>
>
>
> --
>
> Alexander Merck
>
> Duke University
>
> IT Security Office
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20180221/a759ca55/attachment-0002.html>