[Oisf-users] Capture loss ~50% reported using Myricom with Suri v 5.0.2

Fri Jun 19 18:29:00 UTC 2020

Hi Fatema,

That’s interesting because the way I read it from this article

https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom

You could set them on the command line like this…

SNF_NUM_RINGS=16 SNF_DATARING_SIZE=17179869184 SNF_DESCRING_SIZE=4294967296 SNF_FLAGS=0x1 suricata -c suricata.yaml -i eth5 --runmode=workers

I only use Myricom cards on my Zeek servers so I haven’t tested it and it has no such restrictions.

Good luck,

Craig

From: fatema bannatwala <fatema.bannatwala at gmail.com>
Sent: Friday, June 19, 2020 1:16 PM
To: Edgmand, Craig <craig.edgmand at okstate.edu>
Cc: Open Information Security Foundation <oisf-users at lists.openinfosecfoundation.org>
Subject: Re: [Oisf-users] Capture loss ~50% reported using Myricom with Suri v 5.0.2

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Thanks Craig, I tried increasing SNF_DATARING_SIZE, but that variable gets overwritten and controlled by pcap.buffer-size in suricata.yml file which allows a max of 2gb, can't set more than that.
Hence setting SNF_DATARING_SIZE explicitly has no effect since.

This has been done:
The following pull request opened by Myricom in the libpcap project indicates that a future SNF software release could provide support for setting the SNF_DATARING_SIZE via the pcap.buffer-size yaml setting:
Ref: https://github.com/the-tcpdump-group/libpcap/pull/435<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fthe-tcpdump-group%2Flibpcap%2Fpull%2F435&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd4444c14ab184963043408d8147cf441%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637281874192318869&sdata=jhJDwiTawSglR4%2FOm33%2F685%2B2oXHYSnzu0ysXIj0%2BaE%3D&reserved=0>

On Fri, Jun 19, 2020 at 11:09 AM Edgmand, Craig <craig.edgmand at okstate.edu<mailto:craig.edgmand at okstate.edu>> wrote:
Hi Fetema,

Not an expert, but have you tried increasing these

SNF_DATARING_SIZE=4096MB

SNF_DESCRING_SIZE=1024MB

If you have the memory, I would multiply these by a factor of 4.  On my servers these numbers are huge. Might also increase buffer size.

Thanks,

Craig

From: Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org<mailto:oisf-users-bounces at lists.openinfosecfoundation.org>> On Behalf Of fatema bannatwala
Sent: Friday, June 19, 2020 12:59 PM
To: Open Information Security Foundation <oisf-users at lists.openinfosecfoundation.org<mailto:oisf-users at lists.openinfosecfoundation.org>>
Subject: [Oisf-users] Capture loss ~50% reported using Myricom with Suri v 5.0.2

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe
Hello Experts,

Need some help tuning down our prod suricata box running Suricata v 5.0.2 with Myricom NIC: 10G-PCIE-8B-S myri_snf 3.0.20.50894

It is consistently reporting ~50% capture loss, calculated based off of the capture.kernel_packets and capture.kernel_dropped values reported in stats.log file.

I have followed the https://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.inliniac.net%2F2012%2F07%2F10%2Fsuricata-on-myricom-capture-cards%2F&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd4444c14ab184963043408d8147cf441%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637281874192328864&sdata=5KgANusNnHK7cijv1B0rUDomBZaosOK8AuD9mTfd2Dg%3D&reserved=0>
guide to pin the cpus to the worker nodes and use pcap.buffer_size to increase the SNF dataring size, but no effect..

We have one Myri card connected to p2p1 and two NUMA nodes, each with 8 cores (16 HT):
NUMA node0 CPU(s):     0-7,16-23
NUMA node1 CPU(s):     8-15,24-31
OS: Centos 7

Any help in the right direction would be appreciated! :)

Thanks!
Fatema

Following is settings from suricata.yml file

# Myricom support

pcap:

  - interface: p2p1

    threads: 14

    buffer-size: 2gb

    checksum-checks: no

pcap-file:

  checksum-checks: auto

threading:

  set-cpu-affinity: yes

  cpu-affinity:

    - management-cpu-set:

        cpu: [ "0" ]

        mode: "balanced"

        prio:

          default: "low"

    - worker-cpu-set:

        cpu: [ "1-7","9-15" ]

        mode: "exclusive"

        prio:

          default: "high"

Following is the currently recorded stats.log:

------------------------------------------------------------------------------------

Date: 6/19/2020 -- 10:55:36 (uptime: 0d, 04h 04m 10s)

------------------------------------------------------------------------------------

Counter                                       | TM Name                   | Value

------------------------------------------------------------------------------------

capture.kernel_packets                        | Total                     | 28447139411

capture.kernel_drops                          | Total                     | 27910518132

capture.kernel_ifdrops                        | Total                     | 6034

decoder.pkts                                  | Total                     | 536633135

SNF parameters:

SNF_APP_ID=32

SNF_DATARING_SIZE=4096MB

SNF_DESCRING_SIZE=1024MB

SNF_NUM_RINGS=14

SNF_FLAGS=0x1

LD_PRELOAD="/opt/snf/lib/libpcap.so.1"

OPTIONS="--user suricata --group suricata --pcap"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20200619/f3fe8863/attachment-0001.html>