[Oisf-users] EXTERNAL: Fwd: Installing / Running Suricata with Myricom NICs

Mon Feb 26 20:12:44 UTC 2018

Hi Alexander, Erich,

I have noticed significant bursts of drops on our Myricom sensors as well.  

I am wondering if you’ve noticed this as well: During periods of drops, I notice one of the workers is pegged and the others are almost idle. I have bypass enabled too. I am curious if this is an “elephant flow” – a disproportionate amount of traffic being hashed to a single worker via the Myricom RSS. I’m not sure. It eventually seems to resolve itself, but not after significant loss. The strange part to me is how low the usage on the other works is when this happens.

My yaml settings are not quite as generous as Erichs, but I will try tweaking them. 

$ top -p $(pgrep Suricata-Main) -H

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND

28092 suri+  20   0  0.173t 0.172t 0.156t R 99.7 70.2   6125:10 W#02-snf0

28107 suri+  20   0  0.173t 0.172t 0.156t S  1.3 70.2 731:43.64 FM#01

28091 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   3217:08 W#01-snf0

28095 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   3369:32 W#05-snf0

28096 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   3401:27 W#06-snf0

28097 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   6416:49 W#07-snf0

28100 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   2837:30 W#10-snf0

28101 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   7366:56 W#11-snf0

28102 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   2792:57 W#12-snf0

28103 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   3203:58 W#13-snf0

28105 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   3508:24 W#15-snf0

28106 suri+  20   0  0.173t 0.172t 0.156t S  0.7 70.2   6806:13 W#16-snf0

28093 suri+  20   0  0.173t 0.172t 0.156t S  0.3 70.2   3353:10 W#03-snf0

28094 suri+  20   0  0.173t 0.172t 0.156t S  0.3 70.2   4036:36 W#04-snf0

28098 suri+  20   0  0.173t 0.172t 0.156t S  0.3 70.2   2819:50 W#08-snf0

28099 suri+  20   0  0.173t 0.172t 0.156t S  0.3 70.2   3948:24 W#09-snf0

28104 suri+  20   0  0.173t 0.172t 0.156t S  0.3 70.2   3068:59 W#14-snf0

28108 suri+  20   0  0.173t 0.172t 0.156t S  0.3 70.2 339:50.10 FR#01

28071 suri+  20   0  0.173t 0.172t 0.156t S  0.0 70.2  17:07.49 Suricata-Main

28109 suri+  20   0  0.173t 0.172t 0.156t S  0.0 70.2   0:10.36 CW

28110 suri+  20   0  0.173t 0.172t 0.156t S  0.0 70.2   0:25.10 CS

One other note, if you’d like to see the debug info, try setting the SNF_DEBUG_FILENAME to a path (i.e. SNF_DEBUG_FILENAME='/tmp/snf.out' ) in additional to the SNF_DEBUG_MASK. That seems to work for me.

Thanks,

Zach

________________________

Zach Rasmor

Email: zachary.r.rasmor at lmco.com <mailto:zachary.r.rasmor at lmco.com> 

Office: 301.921.7080

From: Oisf-users [mailto:oisf-users-bounces at lists.openinfosecfoundation.org] On Behalf Of Erich Lerch
Sent: Wednesday, February 21, 2018 2:45 AM
To: oisf-users at lists.openinfosecfoundation.org
Subject: EXTERNAL: [Oisf-users] Fwd: Installing / Running Suricata with Myricom NICs

Hi Alex

We seem to have a similar setup as of OS, hardware, traffic and myricom.

We experience almost no drops (usually less than 0.2%).

We start suri with these params:

SNF_NUM_RINGS=10 SNF_FLAGS=0x1 SNF_DATARING_SIZE=12884901888 SNF_DESCRING_SIZE=3221225472

We do start Suri with "-i snf0", it throws a warning in suricata.log, but it works:
<Warning> - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'snf0': No such device (19)

There are other factors which influence the performance:

- I set a BPF to bypass traffic I don't want to see anyway

- number of rules (we have activated about 22'000 rules)

- suricata.yaml, some configs greatly influence performance

Some excerpts:

...
pcap:
  - interface: snf0
    threads: 10   <<--- has to correspond with SNF_NUM_RINGS
    buffer-size: 2gb
    checksum-checks: no
    promisc: no
    snaplen: 1520
    bpf-filter: "..."

...
stream:
  memcap: 5gb
...
    reassembly:
    memcap: 10gb
...

detect:
  profile: custom
  custom-values:
    toclient-groups: 200
    toserver-groups: 200
...
# hyperscan
mpm-algo: hs
spm-algo: hs

# pin to cores

threading:
  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ ... ]
    - worker-cpu-set:
        cpu: [ ... ]
        mode: "exclusive" # run detect threads in these cpus
        threads: 10
        prio:
          high: [ ... ]
          default: "medium"
...
max-pending-packets: 8192
...
flow:
  memcap: 1024mb
  hash-size: 524288
  prealloc: 1048576
  emergency-recovery: 30
  managers: 2
  recyclers: 2
...

- try to pin suri worker threads to the same NUMA node the myricom is attached to

HTH,

erich

2018-02-20 18:58 GMT+01:00 Alexander Merck <alexander.merck at duke.edu <mailto:alexander.merck at duke.edu> >:

Hello,

Hopefully someone can help shed some light on some issues we've been seeing. We just installed a new instance of Suricata on a fresh RHEL7 monitoring box with Myricom cards. However, we are seeing significant packet loss (20-35%) on 2-3 Gbps traffic when attempting to use the SNF drivers.

I'm suspecting that the Myricom SNF drivers are not functioning as expected. We're able to run tcpdump compiled against these drivers with no issue, including generating debug output. We've also found when supplying the SNF_DEBUG_MASK environment variable when running Suricata, no debug output is generated.

Also, when using Suricata with the SNF drivers, should you be able to use the interface names specified by SNF (e.g. snf0)? When trying to run Suricata using the -i snf0, we get an "Unable to find iface snf0: No such device" error message. We are only able to run Suricata against the interface names specified by the kernel (in our case, enp4s0)

The version of Suricata is 4.0.4 and the version of SNF is 3.0.12. Running ldd shows that Suricata is linked against the SNF libraries.

# ldd /usr/bin/suricata

...

        libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007f238ffb0000)

...

        libsnf.so.0 => /opt/snf/lib/libsnf.so.0 (0x00007f238dae4000)

We compiled Suricata per these instructions: https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom. I did notice that this document is over five years old, but all of the configuration options seemed correct.

./configure --with-libpcap-includes=/opt/snf/include/ --with-libpcap-libraries=/opt/snf/lib/ --prefix=/usr --sysconfdir=/etc --localstatedir=/var

And we're running Suricata with the following command:

SNF_NUM_RINGS=32 SNF_DATARING_SIZE=17179869184 SNF_DESCRING_SIZE=4294967296 SNF_FLAGS=0x1 SNF_DEBUG_MASK=3 SNF_DEBUG_FILENAME="/tmp/snf.out" /usr/bin/suricata -c /etc/suricata/suricata.yaml -i enp4s0 --runmode=workers

The box we’re running this on has 64 cores and 256GB of RAM, so I doubt it’s a resource issue…but could potentially be a configuration issue.

Are we missing something in the install process that may be causing these issues? Any recommendations or pointers would be greatly appreciated. Thanks!

-Alex M

-- 

Alexander Merck

Duke University

IT Security Office

_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org <mailto:oisf-users at openinfosecfoundation.org> 
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users

Conference: https://suricon.net
Trainings: https://suricata-ids.org/training/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20180226/2ea4137d/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7456 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20180226/2ea4137d/attachment-0002.bin>