[Oisf-users] EXTERNAL: Fwd: Installing / Running Suricata with Myricom NICs
Erich Lerch
erich.lerch at gmail.com
Mon Feb 26 21:10:00 UTC 2018
Hi Zach
Yes, happens here, too! Fortunately not too often, and only for a short
period of time, before it normalizes again.
Never found out why exactly this happens, though.
Erich
On 26.02.2018 21:12, Rasmor, Zachary R wrote:
> Hi Alexander, Erich,
>
>
>
> I have noticed significant bursts of drops on our Myricom sensors as well.
>
>
>
> I am wondering if you’ve noticed this as well: During periods of drops,
> I notice one of the workers is pegged and the others are almost idle. I
> have bypass enabled too. I am curious if this is an “elephant flow” – a
> disproportionate amount of traffic being hashed to a single worker via
> the Myricom RSS. I’m not sure. It eventually seems to resolve itself,
> but not after significant loss. The strange part to me is how low the
> usage on the other works is when this happens.
>
>
>
> My yaml settings are not quite as generous as Erichs, but I will try
> tweaking them.
>
>
>
> $ top -p $(pgrep Suricata-Main) -H
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> *28092 suri+ 20 0 0.173t 0.172t 0.156t R 99.7 70.2 6125:10 W#02-snf0*
>
> 28107 suri+ 20 0 0.173t 0.172t 0.156t S 1.3 70.2 731:43.64 FM#01
>
> 28091 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3217:08 W#01-snf0
>
> 28095 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3369:32 W#05-snf0
>
> 28096 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3401:27 W#06-snf0
>
> 28097 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 6416:49 W#07-snf0
>
> 28100 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 2837:30 W#10-snf0
>
> 28101 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 7366:56 W#11-snf0
>
> 28102 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 2792:57 W#12-snf0
>
> 28103 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3203:58 W#13-snf0
>
> 28105 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3508:24 W#15-snf0
>
> 28106 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 6806:13 W#16-snf0
>
> 28093 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 3353:10 W#03-snf0
>
> 28094 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 4036:36 W#04-snf0
>
> 28098 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 2819:50 W#08-snf0
>
> 28099 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 3948:24 W#09-snf0
>
> 28104 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 3068:59 W#14-snf0
>
> 28108 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 339:50.10 FR#01
>
> 28071 suri+ 20 0 0.173t 0.172t 0.156t S 0.0 70.2 17:07.49
> Suricata-Main
>
> 28109 suri+ 20 0 0.173t 0.172t 0.156t S 0.0 70.2 0:10.36 CW
>
> 28110 suri+ 20 0 0.173t 0.172t 0.156t S 0.0 70.2 0:25.10 CS
>
>
>
> One other note, if you’d like to see the debug info, try setting the
> SNF_DEBUG_FILENAME to a path (i.e. SNF_DEBUG_FILENAME='/tmp/snf.out' )
> in additional to the SNF_DEBUG_MASK. That seems to work for me.
>
>
>
> Thanks,
>
> Zach
>
> *________________________*
>
> *Zach Rasmor*
>
> Email: zachary.r.rasmor at lmco.com <mailto:zachary.r.rasmor at lmco.com>**
>
> Office: 301.921.7080
>
>
>
> *From:* Oisf-users
> [mailto:oisf-users-bounces at lists.openinfosecfoundation.org] *On Behalf
> Of *Erich Lerch
> *Sent:* Wednesday, February 21, 2018 2:45 AM
> *To:* oisf-users at lists.openinfosecfoundation.org
> *Subject:* EXTERNAL: [Oisf-users] Fwd: Installing / Running Suricata
> with Myricom NICs
>
>
>
> Hi Alex
>
>
>
> We seem to have a similar setup as of OS, hardware, traffic and myricom.
>
> We experience almost no drops (usually less than 0.2%).
>
> We start suri with these params:
>
> SNF_NUM_RINGS=10 SNF_FLAGS=0x1 SNF_DATARING_SIZE=12884901888
> SNF_DESCRING_SIZE=3221225472
>
>
>
> We do start Suri with "-i snf0", it throws a warning in suricata.log,
> but it works:
> <Warning> - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get
> feature via ioctl for 'snf0': No such device (19)
>
>
>
> There are other factors which influence the performance:
>
> - I set a BPF to bypass traffic I don't want to see anyway
>
> - number of rules (we have activated about 22'000 rules)
>
>
>
> - suricata.yaml, some configs greatly influence performance
>
> Some excerpts:
>
> ...
> pcap:
> - interface: snf0
> threads: 10 <<--- has to correspond with SNF_NUM_RINGS
> buffer-size: 2gb
> checksum-checks: no
> promisc: no
> snaplen: 1520
> bpf-filter: "..."
>
> ...
> stream:
> memcap: 5gb
> ...
> reassembly:
> memcap: 10gb
> ...
>
> detect:
> profile: custom
> custom-values:
> toclient-groups: 200
> toserver-groups: 200
> ...
> # hyperscan
> mpm-algo: hs
> spm-algo: hs
>
> # pin to cores
>
> threading:
> set-cpu-affinity: yes
> cpu-affinity:
> - management-cpu-set:
> cpu: [ ... ]
> - worker-cpu-set:
> cpu: [ ... ]
> mode: "exclusive" # run detect threads in these cpus
> threads: 10
> prio:
> high: [ ... ]
> default: "medium"
> ...
> max-pending-packets: 8192
> ...
> flow:
> memcap: 1024mb
> hash-size: 524288
> prealloc: 1048576
> emergency-recovery: 30
> managers: 2
> recyclers: 2
> ...
>
> - try to pin suri worker threads to the same NUMA node the myricom is
> attached to
>
> HTH,
>
> erich
>
>
>
>
>
> 2018-02-20 18:58 GMT+01:00 Alexander Merck <alexander.merck at duke.edu
> <mailto:alexander.merck at duke.edu>>:
>
> Hello,
>
>
>
> Hopefully someone can help shed some light on some issues we've been
> seeing. We just installed a new instance of Suricata on a fresh
> RHEL7 monitoring box with Myricom cards. However, we are seeing
> significant packet loss (20-35%) on 2-3 Gbps traffic when attempting
> to use the SNF drivers.
>
>
>
>
>
> I'm suspecting that the Myricom SNF drivers are not functioning as
> expected. We're able to run tcpdump compiled against these drivers
> with no issue, including generating debug output. We've also found
> when supplying the SNF_DEBUG_MASK environment variable when running
> Suricata, no debug output is generated.
>
>
>
> Also, when using Suricata with the SNF drivers, should you be able
> to use the interface names specified by SNF (e.g. snf0)? When trying
> to run Suricata using the -i snf0, we get an "Unable to find iface
> snf0: No such device" error message. We are only able to run
> Suricata against the interface names specified by the kernel (in our
> case, enp4s0)
>
>
>
>
>
> The version of Suricata is 4.0.4 and the version of SNF is 3.0.12.
> Running ldd shows that Suricata is linked against the SNF libraries.
>
>
>
> # ldd /usr/bin/suricata
>
> ...
>
> libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007f238ffb0000)
>
> ...
>
> libsnf.so.0 => /opt/snf/lib/libsnf.so.0 (0x00007f238dae4000)
>
>
>
>
>
> We compiled Suricata per these instructions:
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom.
> I did notice that this document is over five years old, but all of
> the configuration options seemed correct.
>
>
>
> ./configure --with-libpcap-includes=/opt/snf/include/
> --with-libpcap-libraries=/opt/snf/lib/ --prefix=/usr
> --sysconfdir=/etc --localstatedir=/var
>
>
>
> And we're running Suricata with the following command:
>
>
>
> SNF_NUM_RINGS=32 SNF_DATARING_SIZE=17179869184
> SNF_DESCRING_SIZE=4294967296 SNF_FLAGS=0x1 SNF_DEBUG_MASK=3
> SNF_DEBUG_FILENAME="/tmp/snf.out" /usr/bin/suricata -c
> /etc/suricata/suricata.yaml -i enp4s0 --runmode=workers
>
>
>
> The box we’re running this on has 64 cores and 256GB of RAM, so I
> doubt it’s a resource issue…but could potentially be a configuration
> issue.
>
>
>
>
>
> Are we missing something in the install process that may be causing
> these issues? Any recommendations or pointers would be greatly
> appreciated. Thanks!
>
>
>
> -Alex M
>
>
>
>
>
> --
>
> Alexander Merck
>
> Duke University
>
> IT Security Office
>
>
>
> _______________________________________________
> Suricata IDS Users mailing list:
> oisf-users at openinfosecfoundation.org
> <mailto:oisf-users at openinfosecfoundation.org>
> Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
>
>
>
>
>
More information about the Oisf-users
mailing list