[Oisf-users] Capture loss ~50% reported using Myricom with Suri v 5.0.2
fatema bannatwala
fatema.bannatwala at gmail.com
Mon Jun 22 06:07:42 UTC 2020
So, I printed out the per thread stats, and it looks like the
"capture.kernel_drops" for every thread (14 total) is the same as the "SNF
drop ring full" counter.
And then the Total "capture.kernel_drops" just adds up to 14 x the per
thread "capture.kernel_drops" which seems wrong?
How does Suricata calculate the "capture.kernel_drops" reported in
stats.log?
# /opt/snf/bin/myri_counters
Lanai uptime (seconds): 7060339
Counters uptime (seconds): 18774
Net send KBytes: 0
Net recv KBytes: 1600442598
SNF recv pkts: 1458428259
*SNF drop ring full:* *97865592*
Interrupts: 163442397
Net bad PHY/CRC32 drop: 91
*capture.kernel*_packets | Total |
2880399660
*capture.kernel*_drops | Total |
1370118288
*capture.kernel*_ifdrops | Total |
1512
*capture.kernel*_packets | W#01-p2p1 |
249578109
*capture.kernel*_drops | W#01-p2p1 |
97865592
*capture.kernel*_ifdrops | W#01-p2p1 |
108
*capture.kernel*_packets | W#02-p2p1 |
332169436
*capture.kernel*_drops | W#02-p2p1 |
97865592
*capture.kernel*_ifdrops | W#02-p2p1 |
108
*capture.kernel*_packets | W#03-p2p1 |
154629355
*capture.kernel*_drops | W#03-p2p1 |
97865592
*capture.kernel*_ifdrops | W#03-p2p1 |
108
*capture.kernel*_packets | W#04-p2p1 |
148255255
*capture.kernel*_drops | W#04-p2p1 |
97865592
*capture.kernel*_ifdrops | W#04-p2p1 |
108
*capture.kernel*_packets | W#05-p2p1 |
149307385
*capture.kernel*_drops | W#05-p2p1 |
97865592
*capture.kernel*_ifdrops | W#05-p2p1 |
108
*capture.kernel*_packets | W#06-p2p1 |
177628220
*capture.kernel*_drops | W#06-p2p1 |
97865592
*capture.kernel*_ifdrops | W#06-p2p1 |
108
*capture.kernel*_packets | W#07-p2p1 |
322356666
*capture.kernel*_drops | W#07-p2p1 |
97865592
*capture.kernel*_ifdrops | W#07-p2p1 |
108
*capture.kernel*_packets | W#08-p2p1 |
173429446
*capture.kernel*_drops | W#08-p2p1 |
97865592
*capture.kernel*_ifdrops | W#08-p2p1 |
108
*capture.kernel*_packets | W#09-p2p1 |
156447639
*capture.kernel*_drops | W#09-p2p1 |
97865592
*capture.kernel*_ifdrops | W#09-p2p1 |
108
*capture.kernel*_packets | W#10-p2p1 |
303111334
*capture.kernel*_drops | W#10-p2p1 |
97865592
*capture.kernel*_ifdrops | W#10-p2p1 |
108
*capture.kernel*_packets | W#11-p2p1 |
163892800
*capture.kernel*_drops | W#11-p2p1 |
97865592
*capture.kernel*_ifdrops | W#11-p2p1 |
108
*capture.kernel*_packets | W#12-p2p1 |
195881047
*capture.kernel*_drops | W#12-p2p1 |
97865592
*capture.kernel*_ifdrops | W#12-p2p1 |
108
*capture.kernel*_packets | W#13-p2p1 |
186604216
*capture.kernel*_drops | W#13-p2p1 |
97865592
*capture.kernel*_ifdrops | W#13-p2p1 |
108
*capture.kernel*_packets | W#14-p2p1 |
167108752
*capture.kernel*_drops | W#14-p2p1 |
97865592
*capture.kernel*_ifdrops | W#14-p2p1 |
108
On Fri, Jun 19, 2020 at 11:29 AM Edgmand, Craig <craig.edgmand at okstate.edu>
wrote:
> Hi Fatema,
>
>
>
> That’s interesting because the way I read it from this article
>
>
>
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom
>
>
>
> You could set them on the command line like this…
>
>
>
> SNF_NUM_RINGS=16 SNF_DATARING_SIZE=17179869184
> SNF_DESCRING_SIZE=4294967296 SNF_FLAGS=0x1 suricata -c suricata.yaml -i
> eth5 --runmode=workers
>
>
>
>
>
> I only use Myricom cards on my Zeek servers so I haven’t tested it and it
> has no such restrictions.
>
>
>
> Good luck,
>
>
>
> Craig
>
>
>
> *From:* fatema bannatwala <fatema.bannatwala at gmail.com>
> *Sent:* Friday, June 19, 2020 1:16 PM
> *To:* Edgmand, Craig <craig.edgmand at okstate.edu>
> *Cc:* Open Information Security Foundation <
> oisf-users at lists.openinfosecfoundation.org>
> *Subject:* Re: [Oisf-users] Capture loss ~50% reported using Myricom with
> Suri v 5.0.2
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe
>
> Thanks Craig, I tried increasing SNF_DATARING_SIZE, but that variable gets
> overwritten and controlled by pcap.buffer-size in suricata.yml file which
> allows a max of 2gb, can't set more than that.
>
> Hence setting SNF_DATARING_SIZE explicitly has no effect since.
>
>
>
> This has been done:
>
> The following pull request opened by Myricom in the libpcap project
> indicates that a future SNF software release could provide support for
> setting the SNF_DATARING_SIZE via the pcap.buffer-size yaml setting:
>
> Ref: https://github.com/the-tcpdump-group/libpcap/pull/435
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fthe-tcpdump-group%2Flibpcap%2Fpull%2F435&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd4444c14ab184963043408d8147cf441%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637281874192318869&sdata=jhJDwiTawSglR4%2FOm33%2F685%2B2oXHYSnzu0ysXIj0%2BaE%3D&reserved=0>
>
>
>
>
>
>
>
> On Fri, Jun 19, 2020 at 11:09 AM Edgmand, Craig <craig.edgmand at okstate.edu>
> wrote:
>
> Hi Fetema,
>
>
>
> Not an expert, but have you tried increasing these
>
>
>
> SNF_DATARING_SIZE=4096MB
>
> SNF_DESCRING_SIZE=1024MB
>
>
>
> If you have the memory, I would multiply these by a factor of 4. On my
> servers these numbers are huge. Might also increase buffer size.
>
>
>
> Thanks,
>
>
>
> Craig
>
>
>
> *From:* Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org> *On
> Behalf Of *fatema bannatwala
> *Sent:* Friday, June 19, 2020 12:59 PM
> *To:* Open Information Security Foundation <
> oisf-users at lists.openinfosecfoundation.org>
> *Subject:* [Oisf-users] Capture loss ~50% reported using Myricom with
> Suri v 5.0.2
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe
>
> Hello Experts,
>
>
>
> Need some help tuning down our prod suricata box running Suricata v 5.0.2
> with Myricom NIC: 10G-PCIE-8B-S myri_snf 3.0.20.50894
>
>
>
> It is consistently reporting ~50% capture loss, calculated based off of
> the capture.kernel_packets and capture.kernel_dropped values reported in
> stats.log file.
>
>
>
> I have followed the
> https://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/
> <https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.inliniac.net%2F2012%2F07%2F10%2Fsuricata-on-myricom-capture-cards%2F&data=02%7C01%7Ccraig.edgmand%40okstate.edu%7Cd4444c14ab184963043408d8147cf441%7C2a69c91de8494e34a230cdf8b27e1964%7C0%7C0%7C637281874192328864&sdata=5KgANusNnHK7cijv1B0rUDomBZaosOK8AuD9mTfd2Dg%3D&reserved=0>
>
> guide to pin the cpus to the worker nodes and use pcap.buffer_size to
> increase the SNF dataring size, but no effect..
>
>
>
> We have one Myri card connected to p2p1 and two NUMA nodes, each with 8
> cores (16 HT):
>
> NUMA node0 CPU(s): 0-7,16-23
> NUMA node1 CPU(s): 8-15,24-31
>
> OS: Centos 7
>
>
>
> Any help in the right direction would be appreciated! :)
>
>
>
> Thanks!
>
> Fatema
>
>
>
> Following is settings from suricata.yml file
>
>
>
> # Myricom support
>
> pcap:
>
> - interface: p2p1
>
> threads: 14
>
> buffer-size: 2gb
>
> checksum-checks: no
>
> pcap-file:
>
> checksum-checks: auto
>
>
>
> threading:
>
> set-cpu-affinity: yes
>
> cpu-affinity:
>
> - management-cpu-set:
>
> cpu: [ "0" ]
>
> mode: "balanced"
>
> prio:
>
> default: "low"
>
> - worker-cpu-set:
>
> cpu: [ "1-7","9-15" ]
>
> mode: "exclusive"
>
> prio:
>
> default: "high"
>
>
>
> Following is the currently recorded stats.log:
>
>
> ------------------------------------------------------------------------------------
>
> Date: 6/19/2020 -- 10:55:36 (uptime: 0d, 04h 04m 10s)
>
>
> ------------------------------------------------------------------------------------
>
> Counter | TM Name |
> Value
>
>
> ------------------------------------------------------------------------------------
>
> capture.kernel_packets | Total |
> 28447139411
>
> capture.kernel_drops | Total |
> 27910518132
>
> capture.kernel_ifdrops | Total |
> 6034
>
> decoder.pkts | Total |
> 536633135
>
>
>
>
>
> SNF parameters:
>
>
>
>
>
> SNF_APP_ID=32
>
> SNF_DATARING_SIZE=4096MB
>
> SNF_DESCRING_SIZE=1024MB
>
> SNF_NUM_RINGS=14
>
> SNF_FLAGS=0x1
>
>
>
> LD_PRELOAD="/opt/snf/lib/libpcap.so.1"
>
>
>
> OPTIONS="--user suricata --group suricata --pcap"
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20200621/c6fece6c/attachment-0001.html>
More information about the Oisf-users
mailing list