[Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Michał Purzyński]

I will have more observations before the weekend, I've been running Zeek +
Suricata at the same busy office, same traffic, on two different sensors
1. nsm1 - hardware hashing with the low-entropy key (keep reading why)
2. nsm2 - software hashing with cluster_flow

I'll dig more into the source code tomorrow but from what I remember

1. The symmetric hash is disabled by default and cannot be enabled with
ethtool, without changes to the ethtool Victor proposed them once and they
were rejected. Using the low-entropy key was the solution. I might ping
Intel again about that.

BTW - we do not know if the symmeric hardware hashing handles fragmented
packets correctly, i.e. WHAT is hashed. I'll take a look at the X710 specs.

2. AFAIR the cluster_flow ignores card's hash? That will not help, it
hashes ports as well so it will fail to compute consistent hash for
fragmented packets. Now, I can see, from a quick peek at the code, that
sometimes the HW hash is fetched from the SKB, where it was put by the
card. I need to investigate further when it is used, pretty sure that's
only for the QM demux and not for the cluster_flow.


On Tue, Jul 2, 2019 at 9:51 AM Nelson, Cooper <cnelson at ucsd.edu> wrote:

> Ok that is good to know.  Btw, I’m pretty sure the X710 supports symmetric
> hashing in hardware so you don’t need to set  the low-entropy key.  But of
> course it doesn’t hurt.
>
>
>
> At this point I think the best course of action to fix this for the older
> cards (ixgbe) that don’t allow for ‘sd’ hashing is to simply update
> ‘cluster_flow’ to ignore the rxhash from the card (or zero it out) and
> compute a new hash on the header of the reassembled packets.  So you should
> follow the SEPTun guides for the older cards and  let suricata do the load
> balancing in software on cores isolated from the worker threads.
>
>
>
> Fragmented TCP packets will still be directed to the ‘wrong’ RSS queue,
> but will ultimately be copied to the correct worker thread.  They may
> arrive out-of-order, not sure how much of an issue this is.   Something you
> could do to further test this would be to enable full file
> logging/extraction and look for files tagged as “TRUNCATED” in the eve
> logs.  That’s a ‘red flag’ that the stream tracking isn’t working properly
> for big TCP flows.  Keep in mind you will always see some truncated files
> organically; however if you see the same file being truncated that’s
> indicative of a problem with TCP flow tracking for that particular
> network.
>
>
>
> I’m “stuck” on the old hardware for the time being and I can’t get any of
> the new eBPF or XDP modes working on my system, so it would be great if
> cluster_flow could be updated to handle this edge case.
>
>
>
> Btw, one of the many reasons I’m such a fan of using a zero-trust
> deployment with an explicit proxy is that the proxy will ‘streamline’ dirty
> ‘Net traffic and deliver it to the inside interface fully defragmented and
> in the correct order.  Then you can use the ‘autofp’ runmode and guarantee
> that packets are delivered properly to the worker threads.
>
>
>
> -Coop
>
>
>
> *From:* Michał Purzyński <michalpurzynski1 at gmail.com>
> *Sent:* Monday, July 1, 2019 4:48 PM
> *To:* Nelson, Cooper <cnelson at ucsd.edu>
> *Cc:* Peter Manev <petermanev at gmail.com>; Cloherty, Sean E <
> scloherty at mitre.org>; Eric Urban <eurban at umn.edu>; Open Information
> Security Foundation <oisf-users at lists.openinfosecfoundation.org>
> *Subject:* Re: [Oisf-users] [EXT] Re: Packet loss and increased resource
> consumption after upgrade to 4.1.2 with Rust support
>
>
>
> So here's what I did at one of out offices. Intel X710 are used there
>
>
>
> ethtool -L enp17s0f0 combined 4
>
> ethtool -L enp17s0f0 rxhash on
>
> ethtool -K enp17s0f0 ntuple on
>
>
>
> ethtool -X enp17s0f0 hkey
> 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A
> equal 4
>
>
>
> ethtool -N enp17s0f0 rx-flow-hash tcp4 sd
>
>
>
> The _qm flow has been used in both Zeek and Suricata.
>
>
>
> So far I can see traffic hashed correctly. If you have some ideas how I
> could test this further, please let me know.
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190703/4709c1fd/attachment-0001.html>