[Oisf-users] af_packet and rss queue count

[Peter Manev]

On Fri, Jan 6, 2017 at 11:06 AM, erik clark <philosnef at gmail.com> wrote:
> Peter, did you have any opportunity to look at this? Just curious to hear
> what, if anything, you may have found.

Sure - I will let you know as soon as i have feedback, no problem.
(most likely by the end of this week)

>
> On Tue, Jan 3, 2017 at 8:07 AM, Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Thu, Dec 29, 2016 at 1:27 PM, erik clark <philosnef at gmail.com> wrote:
>> > Sorry, I missed the bottom there. I am using
>> > https://github.com/JustinAzoff/can-i-use-afpacket-fanout to confirm that
>> > flows are properly being assigned.
>> >
>> > Be aware, this does NOT work with ESP traffic. I had seen reverse
>> > udp/tcp
>> > failures and couldn't figure out why, so I pulled pcap for the events
>> > (these
>> > are marked with source/dest information from the afpacket go script),
>> > and
>> > saw that 99.7ish% of the events posted with reverse failures were ESP
>> > traffic. Since I can't do anything with ESP content anyway, I could not
>> > care
>> > less that these are failing, as there is no way to do inspection on it.
>> >
>> > To get udp6/tcp6, you just set them with -N as in the example,
>> > specifying
>> > tcp6 instead of tcp4 and so on.
>> >
>>
>> Ok. Thank you.
>> I will take a look and do some runs and will feedback the findings.
>>
>> > On Thu, Dec 29, 2016 at 6:20 AM, erik clark <philosnef at gmail.com> wrote:
>> >>
>> >> Yes, thats the patch at marc.info in the previous email. Please note
>> >> the
>> >> submitter of that patch. :)
>> >>
>> >> On Thu, Dec 29, 2016 at 4:10 AM, Peter Manev <petermanev at gmail.com>
>> >> wrote:
>> >>>
>> >>>
>> >>>
>> >>> On 28 Dec 2016, at 15:00, erik clark <philosnef at gmail.com> wrote:
>> >>>
>> >>> (patch for ixgbe)
>> >>> http://marc.info/?l=linux-netdev&m=148181173415107&w=2
>> >>>
>> >>> (patch to ixgbe/src/kompat.h since it wont compile on rhel7.3 due to
>> >>> kernel version issues. None of this is pf_ring specific)
>> >>>
>> >>>
>> >>> https://raw.githubusercontent.com/ntop/PF_RING/41b6eb733e295c91375b135d2816dbdd09b4b548/drivers/intel/ixgbe/ixgbe-4.1.5-zc/src/kcompat.h
>> >>>
>> >>>
>> >>> In a different mail you mentioned that there is also a patch from Red
>> >>> Hat
>> >>> with regards to at-packet not working on this kernel version - is this
>> >>> fixed/working now?
>> >>>
>> >>>
>> >>> (ethtool arguments and irq setting)
>> >>>
>> >>> ethtool -C em1 rx-usecs 1 adaptive-rx off
>> >>> ethtool -G em1 rx 4096 tx 4096
>> >>> for x in tso gro lro gso rx tx sg; do ethtool -K em1 $x off; done
>> >>> /opt/src/ixgbe-4.4.6/scripts/set_irq_affinity em1
>> >>> ethtool -X em1 hkey
>> >>>
>> >>> 6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a:6d:5a
>> >>> ethtool -N em1 rx-flow-hash tcp4 sdfn
>> >>> ethtool -N em1 rx-flow-hash udp4 sdfn
>> >>>
>> >>>
>> >>> Can you confirm all flows are good for tcp/udp/4/6?
>> >>> (And how do you confirm it?)
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> I have 64 cores, and set_irq_affinity pulls a full set of 63 rss
>> >>> queues.
>> >>> Thanks!
>> >>>
>> >>> On Sat, Dec 24, 2016 at 1:28 PM, erik clark <philosnef at gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> I will post that on Wednesday when I get back to work. Its 6? ethtool
>> >>>> statements, and 2 ixgbe patches (for rhel7 at least). Anything else
>> >>>> should
>> >>>> be just 1 patch. This is running the 4.4.6 ixgbe released from intel
>> >>>> directly. This has worked for the most recent 3 kernels under RHEL7.
>> >>>>
>> >>>> On Sat, Dec 24, 2016 at 12:51 PM, Peter Manev <petermanev at gmail.com>
>> >>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> > On 24 Dec 2016, at 18:22, erik clark <philosnef at gmail.com> wrote:
>> >>>>> >
>> >>>>> > I have seen several places commenting that you should set the RSS
>> >>>>> > queue to 1. However, when examining af_packet with Bro, a patch
>> >>>>> > released
>> >>>>> > from Redhat for the ixgbe kernel module, and some ethtool
>> >>>>> > tweaking, we have
>> >>>>> > found that (for Bro at least) running a full 63 (we have 54 cores)
>> >>>>> > RSS
>> >>>>> > queues vastly improves performance, and keeps state intact across
>> >>>>> > sessions.
>> >>>>> >
>> >>>>> > Based on this update, which fixes the broken implementation of
>> >>>>> > setting a symmetric hash in the hardware of the card
>> >>>>>
>> >>>>> Can you please share a bit in a bit more detail-
>> >>>>> Which ixgbe/kernel version that is ?
>> >>>>> Which patch is it ?
>> >>>>> What is the ethtool tweaking procedure?
>> >>>>>
>> >>>>> Thanks
>> >>>>>
>> >>>>> > (again ixgbe, not tested with i40e), is it still necessary to run
>> >>>>> > one
>> >>>>> > queue? If so, you can't run Bro and Suri on the same box with
>> >>>>> > af_packet and
>> >>>>> > get equivalent performance out of both tools. Having run Suri with
>> >>>>> > 63 queues
>> >>>>> > for a week now, it seems performance is considerably better than
>> >>>>> > with
>> >>>>> > pf_ring, and I can not find any unusual behavior in my alerts...
>> >>>>> >
>> >>>>> > _______________________________________________
>> >>>>> > Suricata IDS Users mailing list:
>> >>>>> > oisf-users at openinfosecfoundation.org
>> >>>>> > Site: http://suricata-ids.org | Support:
>> >>>>> > http://suricata-ids.org/support/
>> >>>>> > List:
>> >>>>> >
>> >>>>> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> >>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>
>



-- 
Regards,
Peter Manev