[Oisf-users] Are kernel_drops a symptom of dysfunctional afpacket fanout and/or RSS?

[Marshall, Hunter]

Big fan. Long time listener, first time caller.  :-)

Revisiting suricata for new needs. I am experiencing a large number of kernel_drops.

It is likely due to ...
1)  the mysterious failure of my 4.10 kernel to process af_packet fanout.
        - I used https://github.com/JustinAzoff/can-i-use-afpacket-fanout to make that diagnosis.
2)  the presence of multiple receive queues in the vmxnet3 ethernet device (vmware)
       - I MAY have worked around that, see OS/DEVICE DETAILS. I tried setting the indirection table to use one queue. Ummm, …. I think.  :-)

I just do not know whether kernel_drops are the/a symptom of these 2 issues. Details below.

Thx for great tool. Thx for any help

                hunter

STATS.LOG
---------
$ grep -B5 drops /var/log/suricata/stats.log  | tail -6
Date: 8/2/2017 -- 23:14:56 (uptime: 1d, 02h 39m 00s)
-----------------------------------------------------
Counter                   | TM Name            | Value
------------------------------------------------------
capture.kernel_packets   | Total              | 3326654970
capture.kernel_drops     | Total              | 19183864

OS/DEVICE DETAILS
-----------------
(kernel updated via "apt-get install linux-image-generic-hwe-16.04")
16 cpu/32GB RAM running on vmware

$ cat /etc/os-release | head -2
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
$ uname -a
Linux vcasuricatap01 4.10.0-28-generic #32~16.04.2-Ubuntu SMP Thu Jul 20 10:19:48 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
$ ethtool -i ens224
driver: vmxnet3
version: 1.4.a.0-k-NAPI
$ ethtool -n ens224
8 RX rings available
$ ethtool -x ens224
RX flow hash indirection table for ens224 with 8 RX ring(s):
    0:      0     0     0     0     0     0     0     0
    8:      0     0     0     0     0     0     0     0
   16:      0     0     0     0     0     0     0     0
   24:      0     0     0     0     0     0     0     0
RSS hash key:

SURICATA DETAILS (installed via apt-get)
----------------
$ suricata --build-info
This is Suricata version 3.2.3 RELEASE
Features: NFQ PCAP_SET_BUFF LIBPCAP_VERSION_MAJOR=1 AF_PACKET HAVE_PACKET_FANOUT LIBCAP_NG LIBNET1.1 HAVE_HTP_URI_NORMALIZE_HOOK PCRE_JIT HAVE_NSS HAVE_LUA HAVE_LUAJIT HAVE_LIBJANSSON TLS MAGIC
SIMD support: none
Atomic intrisics: 1 2 4 8 byte(s)
64-bits, Little-endian architecture
GCC version 5.4.0 20160609, C version 199901
compiled with _FORTIFY_SOURCE=2
L1 cache line size (CLS)=64
thread local storage method: __thread
compiled with LibHTP v0.5.25, linked against LibHTP v0.5.25

FANOUT TEST
-----------
$ sudo ./can-i-use-afpacket-fanout -interface ens224 2>&1 | grep Final
2017/08/02 23:12:59 Final Stats: packets=1506 flows=101 success_flows=15 failed_flows=63 pkt_success=357 pkt_reverse_success=229 pkt_failures=1048 pkt_reverse_failures=904

RUN DETAILS
-----------
I did not set runmode “workers” explicitly, but the smooth
distribution across all 16 cpus makes me pretty sure that was the setting in play

$ ps aux | grep suricata
root      7749 59.6  0.4 1619732 151048 ?      Ssl  23:24   0:06 /usr/bin/suricata -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid --af-packet -D -vvv --disable-detection

CONFIG FILE DETAIL (af-packet)
-------------------------------
af-packet:
  - interface: ens224
    cluster-id: 99
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes

NETWORK LOAD
--------------
$ sar -n DEV 10 1
Linux 4.10.0-28-generic (vcasuricatap01) 08/02/17    _x86_64_    (16 CPU)

23:30:17        IFACE   rxpck/s   txpck/s    rxkB/s    txkB/s
23:30:27       ens224  53178.80      0.00  44525.42      0.00




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170802/4836de83/attachment-0001.html>