<div dir="ltr"><div>1. af_packet flow defragments before calculating hash</div><div>2. cluster_flow ignores hardware hash and computes it on its own</div><div>3. cluster_qm uses skb_get_queue_mapping() and gets an index of a queue from the hardware (so it uses the hardware hashing, although implictly)<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 3, 2019 at 12:25 AM Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com">michalpurzynski1@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I will have more observations before the weekend, I've been running Zeek + Suricata at the same busy office, same traffic, on two different sensors</div><div>1. nsm1 - hardware hashing with the low-entropy key (keep reading why)</div><div>2. nsm2 - software hashing with cluster_flow</div><div><br></div><div>I'll dig more into the source code tomorrow but from what I remember</div><div><br></div><div>1. The symmetric hash is disabled by default and cannot be enabled with ethtool, without changes to the ethtool Victor proposed them once and they were rejected. Using the low-entropy key was the solution. I might ping Intel again about that.</div><div><br></div><div>BTW - we do not know if the symmeric hardware hashing handles fragmented packets correctly, i.e. WHAT is hashed. I'll take a look at the X710 specs.</div><div><br></div><div>2. AFAIR the cluster_flow ignores card's hash? That will not help, it hashes ports as well so it will fail to compute consistent hash for fragmented packets. Now, I can see, from a quick peek at the code, that sometimes the HW hash is fetched from the SKB, where it was put by the card. I need to investigate further when it is used, pretty sure that's only for the QM demux and not for the cluster_flow.<br></div><div dir="ltr"><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 2, 2019 at 9:51 AM Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-2504437863245195451gmail-m_-2299895818250307732gmail-m_5094504439252820918WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Ok that is good to know. Btw, I’m pretty sure the X710 supports symmetric hashing in hardware so you don’t need to set the low-entropy key. But of course it
doesn’t hurt.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">At this point I think the best course of action to fix this for the older cards (ixgbe) that don’t allow for ‘sd’ hashing is to simply update ‘cluster_flow’ to
ignore the rxhash from the card (or zero it out) and compute a new hash on the header of the reassembled packets. So you should follow the SEPTun guides for the older cards and let suricata do the load balancing in software on cores isolated from the worker
threads.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Fragmented TCP packets will still be directed to the ‘wrong’ RSS queue, but will ultimately be copied to the correct worker thread. They may arrive out-of-order,
not sure how much of an issue this is. Something you could do to further test this would be to enable full file logging/extraction and look for files tagged as “TRUNCATED” in the eve logs. That’s a ‘red flag’ that the stream tracking isn’t working properly
for big TCP flows. Keep in mind you will always see some truncated files organically; however if you see the same file being truncated that’s indicative of a problem with TCP flow tracking for that particular network.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">I’m “stuck” on the old hardware for the time being and I can’t get any of the new eBPF or XDP modes working on my system, so it would be great if cluster_flow
could be updated to handle this edge case. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Btw, one of the many reasons I’m such a fan of using a zero-trust deployment with an explicit proxy is that the proxy will ‘streamline’ dirty ‘Net traffic and
deliver it to the inside interface fully defragmented and in the correct order. Then you can use the ‘autofp’ runmode and guarantee that packets are delivered properly to the worker threads.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">-Coop<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com" target="_blank">michalpurzynski1@gmail.com</a>>
<br>
<b>Sent:</b> Monday, July 1, 2019 4:48 PM<br>
<b>To:</b> Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>><br>
<b>Cc:</b> Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>>; Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>>; Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>>; Open Information Security Foundation <<a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a>><br>
<b>Subject:</b> Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">So here's what I did at one of out offices. Intel X710 are used there<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">ethtool -L enp17s0f0 combined 4<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ethtool -L enp17s0f0 rxhash on<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">ethtool -K enp17s0f0 ntuple on<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">ethtool -X enp17s0f0 hkey 6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A:6D:5A equal 4<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">ethtool -N enp17s0f0 rx-flow-hash tcp4 sd<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The _qm flow has been used in both Zeek and Suricata.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So far I can see traffic hashed correctly. If you have some ideas how I could test this further, please let me know.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</blockquote></div>
</div>
</blockquote></div>