[Oisf-users] Suricata, modern CPU and scheduling. And NUMA.

[Michal Purzynski]

Hey.

I'm wondering if with a modern CPU (Sandy Bridge, Haswell) should I use 
CPU affinity? We're talking 10Gbit/sec here, not an interface but a real 
traffic. Oh, and I have 2 x 8 cores, 128GB RAM, Myricom card.

(model name    : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz)

Yes, I've seen all the HOWTOs about tuning Suricata for 10Gbit/sec. And 
each of them says something else. Meh. That's why I'm looking for your 
comments - I can't sleep, fighting battles in my brain ;)

There are three possible scenarios here:

1. Leave HT enabled, don't touch affinity, leave scheduling to Linux

In this setup Linux sometimes schedules workers on a "virtual" (HT) 
cores. And that is bad, because two workers compete for resources of the 
same physical core. Am I wrong here? I've seen Linux doing that.
Also, cache coherency sucks here. L2 and L3 to the rescue, a bit. And 
migrating thread between cores should invalidate TLB (partially).

2. Disable HT, don't touch affinity, leave scheduling to Linux.

Haven't tried it yet. It should help in theory.

3. Pin threads to physical cores.

But, Suricata uses not just 16 threads for workers (in my setup). There 
are different "management/housekeeping" ones as well.

Should I reserve some cores for them and set affinity that all of them 
can compete for like 1-2 cores, and then pin workers (less of them) to 
what's remaining?

Or maybe pin 16 workers to cores and let the rest float as they wish?