[Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Nelson, Cooper]

Ok that is good to know.  Btw, I’m pretty sure the X710 supports symmetric hashing in hardware so you don’t need to set  the low-entropy key.  But of course it doesn’t hurt.

At this point I think the best course of action to fix this for the older cards (ixgbe) that don’t allow for ‘sd’ hashing is to simply update ‘cluster_flow’ to ignore the rxhash from the card (or zero it out) and compute a new hash on the header of the reassembled packets.  So you should follow the SEPTun guides for the older cards and  let suricata do the load balancing in software on cores isolated from the worker threads.

Fragmented TCP packets will still be directed to the ‘wrong’ RSS queue, but will ultimately be copied to the correct worker thread.  They may arrive out-of-order, not sure how much of an issue this is.   Something you could do to further test this would be to enable full file logging/extraction and look for files tagged as “TRUNCATED” in the eve logs.  That’s a ‘red flag’ that the stream tracking isn’t working properly for big TCP flows.  Keep in mind you will always see some truncated files organically; however if you see the same file being truncated that’s indicative of a problem with TCP flow tracking for that particular network.

I’m “stuck” on the old hardware for the time being and I can’t get any of the new eBPF or XDP modes working on my system, so it would be great if cluster_flow could be updated to handle this edge case.

Btw, one of the many reasons I’m such a fan of using a zero-trust deployment with an explicit proxy is that the proxy will ‘streamline’ dirty ‘Net traffic and deliver it to the inside interface fully defragmented and in the correct order.  Then you can use the ‘autofp’ runmode and guarantee that packets are delivered properly to the worker threads.

-Coop

From: Michał Purzyński <michalpurzynski1 at gmail.com>
Sent: Monday, July 1, 2019 4:48 PM
To: Nelson, Cooper <cnelson at ucsd.edu>
Cc: Peter Manev <petermanev at gmail.com>; Cloherty, Sean E <scloherty at mitre.org>; Eric Urban <eurban at umn.edu>; Open Information Security Foundation <oisf-users at lists.openinfosecfoundation.org>
Subject: Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

So here's what I did at one of out offices. Intel X710 are used there

ethtool -L enp17s0f0 combined 4
ethtool -L enp17s0f0 rxhash on
ethtool -K enp17s0f0 ntuple on

ethtool -X enp17s0f0 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 4

ethtool -N enp17s0f0 rx-flow-hash tcp4 sd

The _qm flow has been used in both Zeek and Suricata.

So far I can see traffic hashed correctly. If you have some ideas how I could test this further, please let me know.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190702/248bd05b/attachment.html>