[Oisf-users] Suricata with Myricom NIC using only one Worker Thread
Greg Grasmehr
greg.grasmehr at caltech.edu
Mon Jul 1 18:46:56 UTC 2019
Incidentally I should mention that the pinned worker cpus are all in the
same numa node as the card's interfaces and the bpf filtered IPs are DNS
servers
On 07/01/19 11:43:13, Greg Grasmehr wrote:
>
> Here is what I have using a Myricom NIC running SNF v3
>
> SNF_NUM_RINGS=12
> SNF_DATARING_SIZE=12884901888
> SNF_DESCRING_SIZE=3221225472
> SNF_FLAGS=0x1
> SNF_DEBUG_MASK=0x40
> SNF_DEBUG_FILENAME="/tmp/snf.out"
>
>
> pcap:
> - interface: snf0
> buffer-size: 1gb
> bpf-filter: not(host 131.215.139.100 or 131.215.9.49 or 131.215.254.100)
> checksum-checks: auto
> threads: 12
> snaplen: 9000
>
> set-cpu-affinity: yes
> cpu-affinity:
> - management-cpu-set:
> cpu: [ 2,4,6 ] # include only these cpus in affinity settings
> prio:
> default: "medium"
> - worker-cpu-set:
> cpu: [ 1,3,5,7,9,11,13,15,17,19,21,23 ]
> mode: "exclusive"
> prio:
> default: "high"
>
> capture.kernel_packets | W#01-snf0 | 3928912475
> capture.kernel_packets | W#02-snf0 | 3929110288
> capture.kernel_packets | W#03-snf0 | 3929426510
> capture.kernel_packets | W#04-snf0 | 3929535277
> capture.kernel_packets | W#05-snf0 | 3929235398
> capture.kernel_packets | W#06-snf0 | 3929513741
> capture.kernel_packets | W#07-snf0 | 3929626205
> capture.kernel_packets | W#08-snf0 | 3929324518
> capture.kernel_packets | W#09-snf0 | 3929316860
> capture.kernel_packets | W#10-snf0 | 3929105902
> capture.kernel_packets | W#11-snf0 | 3928994921
> capture.kernel_packets | W#12-snf0 | 3928984889
>
> On 06/28/19 09:02:20, Fabian Franz wrote:
> > Hi all,
> >
> >
> > I am currently trying to get Suricata to work together with a Myricom card running a Sniffer10G driver. The problems I have seem to be somewhat similar to what Alexander Merck described on this list in Feb 2018 (https://lists.openinfosecfoundation.org/pipermail/oisf-users/2018-February/007790.html) but I could not find an answer to the problem in there and did not want to dig up such an old thread.
> >
> >
> > I have installed the card and the driver on a Ubuntu 18.04 server with 64gigs of RAM and 16 cores (including HT). I followed the instructions here: https://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/ and here: https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricomto build and run suricata.
> >
> > The card seems to be working and when viewing the debug output using one of the snf driver tools everything seems fine. However, no debug output is generated when setting the debug flag while running suricata.
> >
> > Now this wouldn't bother me too much if it wasn't for the stats.log file. This looks like the following:
> >
> >
> > capture.kernel_packets | W#01-ens5 | 53827149
> > capture.kernel_packets | W#02-ens5 | 10
> > capture.kernel_packets | W#03-ens5 | 9
> > capture.kernel_packets | W#04-ens5 | 0
> > capture.kernel_packets | W#05-ens5 | 0
> > capture.kernel_packets | W#06-ens5 | 10
> > capture.kernel_packets | W#07-ens5 | 0
> > capture.kernel_packets | W#08-ens5 | 18
> > capture.kernel_packets | W#09-ens5 | 2
> > capture.kernel_packets | W#10-ens5 | 2
> > capture.kernel_packets | W#11-ens5 | 20
> > capture.kernel_packets | W#12-ens5 | 4
> > capture.kernel_packets | W#13-ens5 | 2
> > capture.kernel_packets | W#14-ens5 | 3
> > capture.kernel_packets | W#15-ens5 | 3
> > capture.kernel_packets | W#16-ens5 | 4
> >
> >
> > Seemingly, only one worker thread is getting a considerable amount of packets while the others are more or less idle. This can also be confirmed when looking at the load of the single threads using htop. Surely this can't be right? Did I miss anything when setting up the driver and/or suricata? Is there a configuration flag or smiliar that I did not set?
> >
> >
> > The traffic I am currently seeing varies between 1 and 6Gbps. Especially when I am seeing more than 3 Gbps, the capture.kernel_drops counter also of W#01 rises pretty quickly to more than 10%.
> >
> >
> > I would be very grateful for any help or hints!
> >
> > Best
> >
> > FabFaeb
> >
> >
> > P.S:
> >
> > My settings look like this:
> >
> > myricom:
> >
> > SNF_NUM_RINGS=16
> > SNF_FLAGS=0x1
> > SNF_DATARING_SIZE=34359738368
> > SNF_DESCRING_SIZE=8589934592
> >
> >
> > suricata.yaml:
> >
> > pcap:
> > - interface: ens5
> > buffer-size: 2048mb
> > checksum-checks: no
> > threads: 16
> > - interface: default
> >
> > cpu-affinity:
> > - management-cpu-set:
> > cpu: [ 0 ]
> > - receive-cpu-set:
> > cpu: [ 0 ]
> > - worker-cpu-set:
> > cpu: [ "1-15" ]
> > mode: "exclusive"
> > prio:
> > default: "high"
> >
> >
>
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> > List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >
> > Conference: https://suricon.net
> > Trainings: https://suricata-ids.org/training/
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/
More information about the Oisf-users
mailing list