[Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Michał Purzyński]

Hi Coop, I might have stumbled upon something of importance for you ;)

There is a function in the ixgbe_main.c

static inline void ixgbe_rx_hash(struct ixgbe_ring *ring,
                 union ixgbe_adv_rx_desc *rx_desc,
                 struct sk_buff *skb)
{
    u16 *rss_type*;

    if (!(ring->netdev->features & NETIF_F_RXHASH))
        return;

    rss_type = le16_to_cpu(rx_desc->wb.lower.lo_dword.hs_rss.pkt_info) &
           IXGBE_RXDADV_RSSTYPE_MASK;

    if (!rss_type)
        return;

    skb_set_hash(skb, le32_to_cpu(rx_desc->wb.lower.hi_dword.rss),
             (IXGBE_RSS_L4_TYPES_MASK & (1ul << rss_type)) ?
             PKT_HASH_TYPE_L4 : *PKT_HASH_TYPE_L3*);
}

No idea how to set it with ethtool (it might not be supported) but what if
you brave it and hardcode the L3 type hash?

Now, depending on your hardware revision, that might or might not work. On
early versions of X520 using RSS caused reordering of packets, on later
versions it worked, same on X540. What's interesting, on early versions of
X520 disabling RSS disables RSS... kind of. Not fully. It is "disabled" but
it can still cause packets reordering

Not to mention this card cannot deal well with packets anywhere close to
1500 bytes, due to internal architecture.

Hack the driver (for fun and testing) or just upgrade to X710.

I just switched to hardware hashing on another sensor and there's no "wrong
thread anymore". Zeek's history field statistics looks much healthier too.
Yeah, I use Zeek to troubleshoot Suri ;)



On Wed, Jul 3, 2019 at 4:14 PM Nelson, Cooper <cnelson at ucsd.edu> wrote:

> Oh, it’s only reassembling a single 1,500 byte packet.  It could even be
> done by the driver in kernel space, then copied to the appropriate RSS
> queue (potentially arriving out-of-order).
>
>
>
> That is great that we have two successful test cases on the X710 cards.
> Now I just need to figure out how fix it on my ixgbe system!
>
>
>
> -Coop
>
>
>
> *From:* Michał Purzyński <michalpurzynski1 at gmail.com>
> *Sent:* Wednesday, July 3, 2019 4:01 PM
> *To:* Nelson, Cooper <cnelson at ucsd.edu>
> *Cc:* Peter Manev <petermanev at gmail.com>; Cloherty, Sean E <
> scloherty at mitre.org>; Eric Urban <eurban at umn.edu>; Open Information
> Security Foundation <oisf-users at lists.openinfosecfoundation.org>
> *Subject:* Re: [Oisf-users] [EXT] Re: Packet loss and increased resource
> consumption after upgrade to 4.1.2 with Rust support
>
>
>
> It would be surprising indeed if such a low-memory card could reassemble
> entire TCP stream (that's potentially huge). All this card has is a small
> buffer, on the order of like 1MB or so, to allow for frames packetization
> (lol) to send ethernet framese over PCI-E packets.
>
>
>
> Another interesting observation - there have been no packets on the wrong
> thread anymore in my X710 setup that hashes only IP src and dst. Again,
> this is according to expectations, but it's nice to have a data point that
> validates those.
>
>
>
>
>
>
>
> On Wed, Jul 3, 2019 at 10:28 AM Nelson, Cooper <cnelson at ucsd.edu> wrote:
>
> Ok this is very good info, thanks.
>
>
>
> My only experience with the X710 cards is reading the documentation, which
> is where I first heard about this issue.  My memory is that the cards (or
> more likely the driver) had an option to “properly” handle fragmented TCP
> packets and recompute the hash after reassembly, however I haven’t been
> able to find it again after reviewing the docs.  It’s possible I misread or
> misremembered it.
>
>
>
> -Coop
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190703/e2d411bc/attachment.html>