<div dir="ltr">How about ignoring layers above 3 and just going with ip src + ip dst? I'm pretty sure I can do that on i40e.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, May 29, 2019 at 12:45 PM Nelson, Cooper <<a href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">HI all, <br>
<br>
I've been investigating the ' tcp.pkt_on_wrong_thread' issue on my system, the counters are very high currently.<br>
<br>
I'm fairly certain I know what the issue is (at least on Intel cards). See the is blog post:<br>
<br>
<a href="http://adrianchadd.blogspot.com/2014/08/receive-side-scaling-figuring-out-how.html" rel="noreferrer" target="_blank">http://adrianchadd.blogspot.com/2014/08/receive-side-scaling-figuring-out-how.html</a><br>
<br>
Pay close attention to this bit....<br>
<br>
>The Intel and Chelsio NICs will hash on all packets that are fragmented by only hashing on the IPv4 details. So, if it's a fragmented TCP or UDP frame, it will hash the first fragment the same as the others - it'll ignore the TCP/UDP details and only hash on the IPv4 frame. This means that all the fragments in a given IP datagram will hash to the same value and thus the same queue.<br>
<br>
>But if there are a mix of fragmented and non-fragmented packets in a given flow - for example, small versus larger UDP frames - then some may be hashed via the IPv4+TCP or IPv4+UDP details and some will just be hashed via the IPv4 details. This means that packets in the same flow will end up being received in different receive queues and thus highly likely be processed out of order.<br>
<br>
The edge case described above is actually very, very common when monitoring live traffic on a big, busy network. Large HTTP downloads will begin with small, unfragmented TCP packets. However, as the receive window increases over time eventually the TCP packets will become fragmented and end up on the wrong thread. You also won't see this on test/simulated traffic unless you deliberately create these packets. <br>
<br>
The easiest fix for this would be to simply force a trivial 'sd' (src->dst) hash on the Intel NIC or within the ixgbe driver. However, ethtool does not seem to allow this for TCP traffic. I'm thinking it might be possible by modifying the driver source code. <br>
<br>
If anyone has any ideas I would appreciate it. <br>
<br>
-Coop<br>
<br>
-----Original Message-----<br>
From: Oisf-users <<a href="mailto:oisf-users-bounces@lists.openinfosecfoundation.org" target="_blank">oisf-users-bounces@lists.openinfosecfoundation.org</a>> On Behalf Of Cloherty, Sean E<br>
Sent: Thursday, February 14, 2019 2:00 PM<br>
To: Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>>; Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>><br>
Cc: Open Information Security Foundation <<a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a>><br>
Subject: Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support<br>
<br>
That also seems to be the case with me regarding high counts on tcp.pkt_on_wrong_thread. I've reverted to 4.0.6 using the same setup and YAML and the stats look much better with no packet loss. I will forward the data. <br>
<br>
Thanks.<br>
<br>
-----Original Message-----<br>
From: Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>> <br>
Sent: Wednesday, February 13, 2019 3:52 PM<br>
To: Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>><br>
Cc: Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>>; Open Information Security Foundation <<a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.openinfosecfoundation.org</a>><br>
Subject: Re: [EXT] Re: [Oisf-users] Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support<br>
<br>
On Fri, Feb 8, 2019 at 6:34 PM Eric Urban <<a href="mailto:eurban@umn.edu" target="_blank">eurban@umn.edu</a>> wrote:<br>
><br>
> Peter, I emailed our config to you directly. I mentioned in my original email that we did test having Rust enabled in 4.1.2 where I explicitly disabled the Rust parsers and still experienced significant packet loss. In that case I added the following config under app-layer.protocols but left the rest of the config the same:<br>
><br>
<br>
<br>
Thank you for sharing all the requested information.<br>
Please find below my observations and some suggestions.<br>
<br>
The good news with 4.1.2:<br>
tcp.pkt_on_wrong_thread | Total | 100<br>
<br>
This is very low (lowest i have seen) for the "tcp.pkt_on_wrong_thread " counter especially with a big run like the shared stats -over 20 days.<br>
Do you mind sharing a bit more info on your NIC (Myricom i think - if I am not mistaken) - driver/version/any specific set up - we are trying to keep a record for that here -<br>
<a href="https://redmine.openinfosecfoundation.org/issues/2725#note-13" rel="noreferrer" target="_blank">https://redmine.openinfosecfoundation.org/issues/2725#note-13</a><br>
<br>
<br>
Observations:<br>
with 4.1.2 this counters seem odd -<br>
capture.kernel_packets | Total<br>
| 16345348068<br>
capture.kernel_drops | Total<br>
| 33492892572<br>
aka you have more kernel_drops than kernel_packets - seems odd.<br>
Which makes me think It maybe a "counter" bug of some sort. Are the NIC driver versions the same on both boxes / same NIC config etc ?<br>
<br>
<br>
Suggestions for the 4.1.2 set up:<br>
Try a run where you disable those (false) and run again to see if any difference (?) :<br>
midstream: true # allow midstream session pickups<br>
async-oneside: true # enable async stream handling<br>
<br>
Thank you<br>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a><br>
_______________________________________________<br>
Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/support/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br>
Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/training/</a></blockquote></div>