[Oisf-devel] reliability of 'cluster_flow' in af_packet and pf_ring

Sat Mar 8 17:13:49 UTC 2014

In the context of Duartes report of #1011, I've been playing with some
validation code for the clustering.

There is some debug code in
https://github.com/inliniac/suricata/pull/880, esp.
https://github.com/inliniac/suricata/commit/939e3e3fd9550916361e3f5ff4c54590d3949bbb

If I run this in workers mode with anything above 1 thread, it fails
almost immediately:

suricata: stream-tcp.c:4180: StreamTcpPacket: Assertion
`!(p->flow->thread_id != tv->thread_id)' failed.
Aborted (core dumped)
# gdb ./src/suricata core
...
Reading symbols from /usr/src/oisf/src/suricata...done.

warning: core file may not match specified executable file.
[New LWP 3711]
[New LWP 3712]
[New LWP 3714]
[New LWP 3713]
[New LWP 3710]
[New LWP 3709]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./src/suricata -c copy-of-suricata-pfring.yaml
--af-packet=eth2 -S /dev/null -v'.
Program terminated with signal 6, Aborted.
#0  0x00007f82f353e425 in __GI_raise (sig=<optimized out>) at
../nptl/sysdeps/unix/sysv/linux/raise.c:64
64      ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) f 4
#4  0x0000000000518ec3 in StreamTcpPacket (tv=0x226c33f0, p=0x2deae00,
stt=0x7f82e00138c0, pq=0x226c37a0) at stream-tcp.c:4180
4180                BUG_ON(p->flow->thread_id != tv->thread_id);
(gdb) print p->flow->thread_id
$1 = 3710
(gdb) print tv->thread_id
$2 = 3711
(gdb)

AF_PACKET is set to do cluster_flow. None of the AF_PACKET options seem
to affect it.

# ethtool -k eth2
Offload parameters for eth2:
rx-checksumming: off
tx-checksumming: off
scatter-gather: off
tcp-segmentation-offload: off
udp-fragmentation-offload: off
generic-segmentation-offload: off
generic-receive-offload: off
large-receive-offload: off
rx-vlan-offload: off
tx-vlan-offload: off
ntuple-filters: off
receive-hashing: off

In autofp mode it does work correctly. There we have our own flow
distribution logic instead of relying on the nic/driver/capture method...

Thoughts?

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

-------------- next part --------------
[2529104.780643] dca service started, version 1.12.1
[2529104.782226] Intel(R) 10 Gigabit PCI Express Network Driver - version 3.17.3
[2529104.782228] Copyright (c) 1999-2013 Intel Corporation.
[2529104.782424] ixgbe: Receive-Side Scaling (RSS) set to 1
[2529104.782427] ixgbe: 0000:0a:00.0: ixgbe_check_options: FCoE Offload feature enabled
[2529104.782429] ixgbe: allow_unsupported_sfp Enabled
[2529104.949289] ixgbe 0000:0a:00.0: irq 47 for MSI/MSI-X
[2529104.949297] ixgbe 0000:0a:00.0: irq 48 for MSI/MSI-X
[2529104.951443] ixgbe 0000:0a:00.0: (PCI Express:2.5GT/s:Width x8) 90:e2:xxxxx
[2529104.951522] ixgbe 0000:0a:00.0: eth2: MAC: 2, PHY: 9, SFP+: 3, PBA No: E68793-005
[2529104.951525] ixgbe 0000:0a:00.0: PCI-Express bandwidth of 16GT/s available (PCI Express:2.5GT/s:Width x8)
[2529104.951527] ixgbe 0000:0a:00.0: This is not sufficient for optimal performance of this card.
[2529104.951530] ixgbe 0000:0a:00.0: For optimal performance, at least 20GT/s of bandwidth is required.
[2529104.951532] ixgbe 0000:0a:00.0: A slot with more lanes and/or higher speed is suggested.
[2529104.951540] ixgbe 0000:0a:00.0: eth2: Enabled Features: RxQ: 1 TxQ: 1 RSC 
[2529104.953386] ixgbe 0000:0a:00.0: eth2: Intel(R) 10 Gigabit Network Connection
[2529104.953629] ixgbe: Receive-Side Scaling (RSS) set to 1
[2529104.953633] ixgbe: 0000:0a:00.1: ixgbe_check_options: FCoE Offload feature enabled
[2529104.953635] ixgbe: allow_unsupported_sfp Enabled
[2529104.980039] ixgbe 0000:0a:00.1: (unregistered net_device): WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.
[2529105.090735] IPv6: ADDRCONF(NETDEV_UP): eth2: link is not ready
[2529105.096891] device eth2 entered promiscuous mode
[2529105.122474] ixgbe 0000:0a:00.1: irq 49 for MSI/MSI-X
[2529105.122484] ixgbe 0000:0a:00.1: irq 50 for MSI/MSI-X
[2529105.155248] ixgbe 0000:0a:00.0: eth2: detected SFP+: 3
[2529105.329431] ixgbe 0000:0a:00.1: (PCI Express:2.5GT/s:Width x8) 90:e2:xxxx
[2529105.329509] ixgbe 0000:0a:00.1: eth3: MAC: 2, PHY: 14, SFP+: 6, PBA No: E68793-005
[2529105.329513] ixgbe 0000:0a:00.1: PCI-Express bandwidth of 16GT/s available (PCI Express:2.5GT/s:Width x8)
[2529105.329515] ixgbe 0000:0a:00.1: This is not sufficient for optimal performance of this card.
[2529105.329518] ixgbe 0000:0a:00.1: For optimal performance, at least 20GT/s of bandwidth is required.
[2529105.329520] ixgbe 0000:0a:00.1: A slot with more lanes and/or higher speed is suggested.
[2529105.329524] ixgbe 0000:0a:00.1: eth3: Enabled Features: RxQ: 1 TxQ: 1 RSC 
[2529105.331396] ixgbe 0000:0a:00.1: eth3: Intel(R) 10 Gigabit Network Connection
[2529105.388729] ixgbe 0000:0a:00.0: eth2: detected SFP+: 3
[2529105.620655] ixgbe 0000:0a:00.0: eth2: detected SFP+: 3
[2529105.670776] IPv6: ADDRCONF(NETDEV_UP): eth3: link is not ready
[2529105.673184] device eth3 entered promiscuous mode
[2529105.677205] ixgbe 0000:0a:00.1: eth3: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.
[2529105.732839] ixgbe 0000:0a:00.1: eth3: detected SFP+: 6
[2529105.909288] ixgbe 0000:0a:00.1: eth3: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.
[2529105.972047] ixgbe 0000:0a:00.1: eth3: detected SFP+: 6
[2529106.149801] ixgbe 0000:0a:00.1: eth3: WARNING: Intel (R) Network Connections are quality tested using Intel (R) Ethernet Optics. Using untested modules is not supported and may cause unstable operation or damage to the module or the adapter. Intel Corporation is not responsible for any harm caused by using untested modules.
[2529106.212024] ixgbe 0000:0a:00.1: eth3: detected SFP+: 6
[2529106.660038] ixgbe 0000:0a:00.0: eth2: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[2529106.661978] IPv6: ADDRCONF(NETDEV_CHANGE): eth2: link becomes ready
[2529107.744045] ixgbe 0000:0a:00.1: eth3: NIC Link is Up 10 Gbps, Flow Control: RX/TX
[2529107.746041] IPv6: ADDRCONF(NETDEV_CHANGE): eth3: link becomes ready