[Oisf-users] Suricata, modern CPU and scheduling. And NUMA.

[Anoop Saldanha]

On Sat, Nov 1, 2014 at 7:04 AM, Michal Purzynski <michal at rsbac.org> wrote:
> Hey.
>
> I'm wondering if with a modern CPU (Sandy Bridge, Haswell) should I use CPU
> affinity? We're talking 10Gbit/sec here, not an interface but a real
> traffic. Oh, and I have 2 x 8 cores, 128GB RAM, Myricom card.
>
> (model name    : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz)
>
> Yes, I've seen all the HOWTOs about tuning Suricata for 10Gbit/sec. And each
> of them says something else. Meh. That's why I'm looking for your comments -
> I can't sleep, fighting battles in my brain ;)
>
> There are three possible scenarios here:
>
> 1. Leave HT enabled, don't touch affinity, leave scheduling to Linux
>
> In this setup Linux sometimes schedules workers on a "virtual" (HT) cores.
> And that is bad, because two workers compete for resources of the same
> physical core. Am I wrong here? I've seen Linux doing that.
> Also, cache coherency sucks here. L2 and L3 to the rescue, a bit. And
> migrating thread between cores should invalidate TLB (partially).
>
> 2. Disable HT, don't touch affinity, leave scheduling to Linux.
>
> Haven't tried it yet. It should help in theory.
>
> 3. Pin threads to physical cores.
>
> But, Suricata uses not just 16 threads for workers (in my setup). There are
> different "management/housekeeping" ones as well.
>
> Should I reserve some cores for them and set affinity that all of them can
> compete for like 1-2 cores, and then pin workers (less of them) to what's
> remaining?
>
> Or maybe pin 16 workers to cores and let the rest float as they wish?

I think affinity is definitely a big concern with numa along with
memory allocation.

>From where I see, code should be explicitly targeted for this
architecture, so that code and data lies on the same socket.  I don't
want code on node1 trying to access memory from node2.  I would
duplicate all my tables for all the threads from socket1, allocate
packet bufs separately for each socket, etc.  If there's someone with
an explicity background writing code for numa, I would definitely like
to hear from them.

-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------