<div dir="ltr"><div>Hi Coop, I might have stumbled upon something of importance for you ;)</div><div><br></div><div>There is a function in the ixgbe_main.c</div><div><br></div><div>static inline void ixgbe_rx_hash(struct ixgbe_ring *ring,<br> union ixgbe_adv_rx_desc *rx_desc,<br> struct sk_buff *skb)<br>{<br> u16 <b>rss_type</b>;<br><br> if (!(ring->netdev->features & NETIF_F_RXHASH))<br> return;<br><br> rss_type = le16_to_cpu(rx_desc->wb.lower.lo_dword.hs_rss.pkt_info) &<br> IXGBE_RXDADV_RSSTYPE_MASK;<br><br> if (!rss_type)<br> return;<br><br> skb_set_hash(skb, le32_to_cpu(rx_desc->wb.lower.hi_dword.rss),<br> (IXGBE_RSS_L4_TYPES_MASK & (1ul << rss_type)) ?<br> PKT_HASH_TYPE_L4 : <b>PKT_HASH_TYPE_L3</b>);<br>}</div><div><br></div><div>No idea how to set it with ethtool (it might not be supported) but what if you brave it and hardcode the L3 type hash?</div><div><br></div><div>Now, depending on your hardware revision, that might or might not work. On early versions of X520 using RSS caused reordering of packets, on later versions it worked, same on X540. What's interesting, on early versions of X520 disabling RSS disables RSS... kind of. Not fully. It is "disabled" but it can still cause packets reordering</div><div><br></div><div>Not to mention this card cannot deal well with packets anywhere close to 1500 bytes, due to internal architecture.</div><div><br></div><div>Hack the driver (for fun and testing) or just upgrade to X710.</div><div><br></div><div>I just switched to hardware hashing on another sensor and there's no "wrong thread anymore". Zeek's history field statistics looks much healthier too.</div><div>Yeah, I use Zeek to troubleshoot Suri ;)<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jul 3, 2019 at 4:14 PM Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-3332918111882078271WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Oh, it’s only reassembling a single 1,500 byte packet. It could even be done by the driver in kernel space, then copied to the appropriate RSS queue (potentially
arriving out-of-order). <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">That is great that we have two successful test cases on the X710 cards. Now I just need to figure out how fix it on my ixgbe system!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">-Coop<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Michał Purzyński <<a href="mailto:michalpurzynski1@gmail.com" target="_blank">michalpurzynski1@gmail.com</a>>
<br>
<b>Sent:</b> Wednesday, July 3, 2019 4:01 PM<br>
<b>To:</b> Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>><br>
<b>Cc:</b> Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>>; Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>>; Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>>; Open Information Security Foundation <<a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a>><br>
<b>Subject:</b> Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">It would be surprising indeed if such a low-memory card could reassemble entire TCP stream (that's potentially huge). All this card has is a small buffer, on the order of like 1MB or so, to allow for frames packetization (lol) to send ethernet
framese over PCI-E packets.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Another interesting observation - there have been no packets on the wrong thread anymore in my X710 setup that hashes only IP src and dst. Again, this is according to expectations, but it's nice to have a data point that validates those.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Jul 3, 2019 at 10:28 AM Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Ok this is very good info, thanks.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">My only experience with the X710 cards is reading the documentation, which is where I first heard about
this issue. My memory is that the cards (or more likely the driver) had an option to “properly” handle fragmented TCP packets and recompute the hash after reassembly, however I haven’t been able to find it again after reviewing the docs. It’s possible I
misread or misremembered it.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">-Coop</span><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote></div>