[Oisf-users] packet on wrong thread x710 cards

Mon Jul 6 18:01:40 UTC 2020

There might be several reasons for this - like fragmented packets or
protocols your card does not recognize. The fact that enabling autofp fixes
it, makes me think it's the former. Autofp will re-hash packets after
defragmenting.

Fragmented packets do not have port numbers, but only an indicator "hey I'm
part of flow X" and they will be hashed in a wrong way. You want to avoid
hashing port numbers and only use src + dst IP and maybe the protocol
number, AKA the 3-tuple. Same goes for any packet broker if you have one in
front of your cluster.

Here's my current working setup, systemd configures interfaces here but you
should be able to find what you need.

Look at lines like

/usr/bin/ethtool -N enp8s0f0 rx-flow-hash tcp4 sd

[Unit]
Description=Configure monitor interfaces
Before=bro.service
Wants=network.target

[Service]
Type=oneshot

ExecStart=/usr/bin/ip link set enp8s0f0 promisc on arp off up

# While the interface is still down
ExecStartPost=/usr/bin/ip link set enp8s0f0 mtu 9000
ExecStartPost=-/usr/bin/ethtool -G enp8s0f0 rx 512

ExecStartPost=-/usr/bin/ethtool -L enp8s0f0 combined 6

ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 rx off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 tx off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 tso off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 ufo off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 gso off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 gro off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 lro off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 rxhash on
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 ntuple on
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 sg off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 txvlan off
ExecStartPost=-/usr/bin/ethtool -K enp8s0f0 rxvlan off
ExecStartPost=-/usr/bin/ethtool -X enp8s0f0 hkey
6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A
equal 6
ExecStartPost=-/usr/bin/ethtool -N enp8s0f0 rx-flow-hash tcp4 sd
ExecStartPost=-/usr/bin/ethtool -N enp8s0f0 rx-flow-hash tcp6 sd
ExecStartPost=-/usr/bin/ethtool -N enp8s0f0 rx-flow-hash udp4 sd
ExecStartPost=-/usr/bin/ethtool -N enp8s0f0 rx-flow-hash udp6 sd

ExecStartPost=-/usr/bin/ethtool -C enp8s0f0 adaptive-rx on rx-usecs 64

ExecStartPost=-/usr/bin/ethtool -G enp8s0f0 rx 512

ExecStartPost=-/usr/bin/ethtool -A enp8s0f0 rx off tx off

ExecStartPost=-/usr/bin/sysctl -w sys.net.ipv6.conf.enp8s0f0.disable_ipv6=1

ExecStop=/usr/bin/ip link set enp8s0f0 promisc off arp off down

RemainAfterExit=yes

SuccessExitStatus=1

[Install]
WantedBy=multi-user.target

On Mon, Jul 6, 2020 at 10:06 AM mohammad kashif <kashif.alig at gmail.com>
wrote:

> Hi
>
> I am aware of https://redmine.openinfosecfoundation.org/issues/2725 and
> there seems to be a conclusion that cluster_qm with symetric hashing solved
> packet_on_wrong thread issue. Unformutaly this is not the case for my
> setup.
> I am using two X710 10G cards on two numa nodes with two Intel 5218 CPU HT
> enabled.
> It's going to be a production suricata setup and I am getting around 3-5
> Gbps on one interface. I have enabled only around 6000 rules for testing.
> The only way I don't get any pkt_on_wrong_thread if I use autofp but cpu
> usage goes on the top so I don't think it is sustainable.
>
> I am testing with cluster_qm and symmetric hashing
>
> My setup is
>
> ethtool -i ens3f0
>
> driver: i40e
>
> version: 2.3.2-k
>
> firmware-version: 7.10 0x800075df 19.5.12
>
> expansion-rom-version:
>
> bus-info: 0000:3b:00.0
>
> supports-statistics: yes
>
> supports-test: yes
>
> supports-eeprom-access: yes
>
> supports-register-dump: yes
>
>
> suricata from debian repo
>
> suricata -V
>
> This is Suricata version 4.1.2 RELEASE
>
>
> kernel : uname -r
>
> 4.19.0-9-amd64
>
>
> cat /sys/devices/system/node/node0/cpulist
>
>
> 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
>
>
> cat /sys/devices/system/node/node1/cpulist
>
>
> 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
>
>
> ethtool -L ens3f0 combined 32
>
> ethtool -K ens3f0 rxhash on
>
> ethtool -K ens3f0 ntuple on
>
>
> same with ens4f0
>
>
> ./set_irq_affinity
> 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38,40,42,44,46,48,50,52,54,56,58,60,62
> ens3f0
>
> ./set_irq_affinity
> 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39,41,43,45,47,49,51,53,55,57,59,61,63
> ens4f0
>
>
> ethtool -X ens3f0 hkey
> 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A
> equal 32
>
>
> same with ens4f0
>
>
> for both interfaces
>
> for i in rx tx tso gso gro   rxhash ntuple sg txvlan rxvlan ; do ethtool
> -K ens4f0 $i off ; echo $i ; done
>
>
> for proto in tcp4 udp4 tcp6 udp6; do /sbin/ethtool -N ens4f0 rx-flow-hash
> $proto sdfn ; done
>
>
>
> suricata --dump-config | grep af-packet
>
> af-packet = (null)
>
> af-packet.0 = interface
>
> af-packet.0.interface = ens3f0
>
> af-packet.0.threads = 32
>
> af-packet.0.cluster-id = 99
>
> af-packet.0.cluster-type = cluster_qm
>
> af-packet.0.defrag = yes
>
> af-packet.0.use-mmap = yes
>
> af-packet.0.mmap-locked = yes
>
> af-packet.0.tpacket-v3 = yes
>
> af-packet.0.ring-size = 200000
>
> af-packet.0.block-size = 1048576
>
> af-packet.1 = interface
>
> af-packet.1.interface = ens4f0
>
> af-packet.1.threads = 32
>
> af-packet.1.cluster-id = 98
>
> af-packet.1.cluster-type = cluster_qm
>
> af-packet.1.defrag = yes
>
> af-packet.1.tpacket-v3 = yes
>
> af-packet.1.use-mmap = yes
>
> af-packet.1.mmap-locked = yes
>
> af-packet.1.ring-size = 200000
>
> af-packet.1.block-size = 1048576
>
>
>
> When I start suricata, pkt_on_wrong_thread is around 20 percent of
> capture.kernel_packets but gradually in a few hours it comes down to 1-2%
> but keeps increasing.
>
>
> I can see from pidstat that only even numbered cpu being used on ens3f0
> and odd numbered on ens4fo as expected due to numa node architecture.
>
>
> mpstat shows all cpu being used but the usage is really low, 2-3%.
>
>
> I haven't enabled cpu_affinity in config files as I can not see load an
> issue here.
>
>
> Even if i use a single interface, it is still showing pkt_on_wrong_thread.
>
>
> Any suggestion would be really appreciated as no config change is removing
> pkt_on_wrong_thread.
>
>
>
>
> Regards
>
> Kashif
>
> _______________________________________________
> NOTE: this list will soon be closed. New topics should be brought to:
> https://forum.suricata.io
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20200706/5f4425e2/attachment-0001.html>