[Oisf-users] suricata 3.2.0 for 10Gb performance

[Cooper F. Nelson]

From a purely HPC perspective, Michal's approach is (probably) the
correct one.

The design pattern is something called a 'thread pool', which involves
creating a static pool of threads (ideally pinned to CPU cores) to
maintain cpu affinity and cache coherency.

This in turn requires a supervising program to direct work to the
individual threads.  In the RSS model this role is assigned per-thread
by the kernel, which results in inefficiencies due to context switching
and hardware interrupts.  A generic overview here:

> https://en.wikipedia.org/wiki/Thread_pool

Michal's approach is to 'burn' a single core to handle all interrupts,
epf filters and software hashing in a single supervisor thread.  This
should be provably optimal for most workloads, however if that single
core becomes the bottleneck, all is lost (see Amdahl's Law).

If you are dealing with jumbo frames and an "Elephant Stampede", then
RSS might be a better solution as the worst-case will be dropping
packets for one thread, vs. everything.  This is also a more likely
scenario for HPC networks vs. general ISP traffic.

What I've been experimenting with is a hybrid solution.  I'm using 4k
PCI reads, hardware offloading, a 4k rx buffer, max. irq coalescing and
big buffers to move as many packets as possible per context switch.  I
also renice all suricata threads to -19.

Keep in mind this is hardware dependent (from my warm_boot.sh script):

> #Set 4k PCI reads
> setpci -v -d 8086:10fb e6.b=2e
> 
> ifconfig eth2 up
> ifconfig eth2 mtu 9000

> #set irq affinity
> /usr/local/sbin/set_irq_affinity.sh all eth2

> #set RSS parameters
> ethtool -K eth2 rx-all on
> ethtool -K eth2 gro on
> ethtool -K eth2 lro off
> ethtool -C eth2 adaptive-rx on
> ethtool -C eth2 rx-usecs 1022
> ethtool -G eth2 rx 4096
> ethtool -K eth2 ntuple on
> ethtool -K eth2 rxhash on
> ethtool -X eth2 hkey 65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da:65:da
> ethtool -N eth2 rx-flow-hash udp4 sdfn
> ethtool -N eth2 rx-flow-hash udp6 sdfn
> ethtool -N eth2 rx-flow-hash tcp4 sdfn
> ethtool -N eth2 rx-flow-hash tcp6 sdfn

(start suri here)

> echo "Sleeping for 30 seconds before re-nicing suricata..."
> sleep 30
> renice -n -19 -p $(ps --no-heading -Lo tid $(pgrep -f suri))



On 2/15/2017 7:07 AM, erik clark wrote:
> Ok, so word back from RH is that hashing is not likely to be in RHEL
> anytime soon. They mentioned this as a possible alternative:
> 
> ethtool -K <ingress nic> rxhash off
> 
> They mentioned that there would be a performance penalty as the kernel
> will compute hashes for each skb on ingress, bit it is unlikely to be
> that noticeable.
> 
> IMO, I think specifying a symmetric hash with -X might be preferential
> to turning rxhash off entirely, when dealing with af_packet in rhel7.
> Suggestions?
> 

If you are interested in doing an RSS suricata build, use a source-based
dist. and patch the ixgbe driver yourself if you are using it.  I'll
recommend hardened gentoo:

> https://wiki.gentoo.org/wiki/Project:Hardened

In general I'm recommending Michal's/Peter's method unless you have to
use RSS.

-- 
Cooper Nelson
Network Security Analyst
UCSD ITS Security Team
cnelson at ucsd.edu x41042

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170215/2583c9ab/attachment-0002.sig>