[Oisf-users] Suricata with Myricom NIC using only one Worker Thread

[Greg Grasmehr]

Here is what I have using a Myricom NIC running SNF v3

SNF_NUM_RINGS=12
SNF_DATARING_SIZE=12884901888
SNF_DESCRING_SIZE=3221225472
SNF_FLAGS=0x1
SNF_DEBUG_MASK=0x40
SNF_DEBUG_FILENAME="/tmp/snf.out"


pcap:
  - interface: snf0
    buffer-size: 1gb
    bpf-filter: not(host 131.215.139.100 or 131.215.9.49 or 131.215.254.100)
    checksum-checks: auto
    threads: 12
    snaplen: 9000

  set-cpu-affinity: yes
  cpu-affinity:
    - management-cpu-set:
        cpu: [ 2,4,6 ]  # include only these cpus in affinity settings
        prio:
          default: "medium"
    - worker-cpu-set:
        cpu: [ 1,3,5,7,9,11,13,15,17,19,21,23 ]
        mode: "exclusive"
        prio:
         default: "high"

capture.kernel_packets                     | W#01-snf0                 | 3928912475
capture.kernel_packets                     | W#02-snf0                 | 3929110288
capture.kernel_packets                     | W#03-snf0                 | 3929426510
capture.kernel_packets                     | W#04-snf0                 | 3929535277
capture.kernel_packets                     | W#05-snf0                 | 3929235398
capture.kernel_packets                     | W#06-snf0                 | 3929513741
capture.kernel_packets                     | W#07-snf0                 | 3929626205
capture.kernel_packets                     | W#08-snf0                 | 3929324518
capture.kernel_packets                     | W#09-snf0                 | 3929316860
capture.kernel_packets                     | W#10-snf0                 | 3929105902
capture.kernel_packets                     | W#11-snf0                 | 3928994921
capture.kernel_packets                     | W#12-snf0                 | 3928984889

On 06/28/19 09:02:20, Fabian Franz wrote:
> Hi all, 
> 
> 
> I am currently trying to get Suricata to work together with a Myricom card running a Sniffer10G driver. The problems I have seem to be somewhat similar to what Alexander Merck described on this list in Feb 2018 (https://lists.openinfosecfoundation.org/pipermail/oisf-users/2018-February/007790.html) but I could not find an answer to the problem in there and did not want to dig up such an old thread. 
> 
> 
> I have installed the card and the driver on a Ubuntu 18.04 server with 64gigs of RAM and 16 cores (including HT). I followed the instructions here: https://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/ and here: https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricomto build and run suricata. 
> 
> The card seems to be working and when viewing the debug output using one of the snf driver tools everything seems fine. However, no debug output is generated when setting the debug flag while running suricata. 
> 
> Now this wouldn't bother me too much if it wasn't for the stats.log file. This looks like the following:
> 
> 
> capture.kernel_packets                        | W#01-ens5                 | 53827149
> capture.kernel_packets                        | W#02-ens5                 | 10
> capture.kernel_packets                        | W#03-ens5                 | 9
> capture.kernel_packets                        | W#04-ens5                 | 0
> capture.kernel_packets                        | W#05-ens5                 | 0
> capture.kernel_packets                        | W#06-ens5                 | 10
> capture.kernel_packets                        | W#07-ens5                 | 0
> capture.kernel_packets                        | W#08-ens5                 | 18
> capture.kernel_packets                        | W#09-ens5                 | 2
> capture.kernel_packets                        | W#10-ens5                 | 2
> capture.kernel_packets                        | W#11-ens5                 | 20
> capture.kernel_packets                        | W#12-ens5                 | 4
> capture.kernel_packets                        | W#13-ens5                 | 2
> capture.kernel_packets                        | W#14-ens5                 | 3
> capture.kernel_packets                        | W#15-ens5                 | 3
> capture.kernel_packets                        | W#16-ens5                 | 4
> 
> 
> Seemingly, only one worker thread is getting a considerable amount of packets while the others are more or less idle. This can also be confirmed when looking at the load of the single threads using htop. Surely this can't be right? Did I miss anything when setting up the driver and/or suricata? Is there a configuration flag or smiliar that I did not set?
> 
> 
> The traffic I am currently seeing varies between 1 and 6Gbps. Especially when I am seeing more than 3 Gbps, the capture.kernel_drops counter also of W#01 rises pretty quickly to more than 10%.
> 
> 
> I would be very grateful for any help or hints!
> 
> Best
> 
> FabFaeb 
> 
> 
> P.S: 
> 
> My settings look like this:
> 
> myricom:
> 
> SNF_NUM_RINGS=16
> SNF_FLAGS=0x1
> SNF_DATARING_SIZE=34359738368
> SNF_DESCRING_SIZE=8589934592
> 
> 
> suricata.yaml:
> 
> pcap:
>   - interface: ens5
>     buffer-size: 2048mb
>     checksum-checks: no
>     threads: 16
>   - interface: default
> 
>   cpu-affinity:
>     - management-cpu-set:
>         cpu: [ 0 ] 
>     - receive-cpu-set:
>         cpu: [ 0 ]  
>     - worker-cpu-set:
>         cpu: [ "1-15" ] 
>         mode: "exclusive"
>         prio:
>           default: "high"
> 
> 

> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> 
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/