[Oisf-users] Suricata, modern CPU and scheduling. And NUMA.

[Peter Manev]

On Sat, Nov 1, 2014 at 2:34 AM, Michal Purzynski <michal at rsbac.org> wrote:
> Hey.
>
> I'm wondering if with a modern CPU (Sandy Bridge, Haswell) should I use CPU
> affinity? We're talking 10Gbit/sec here, not an interface but a real
> traffic. Oh, and I have 2 x 8 cores, 128GB RAM, Myricom card.

You could look below for tips/info how to do a set up with Myricom:
http://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/
http://blogs.emulex.com/implementers/tag/myricom/

>
> (model name    : Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz)
>
> Yes, I've seen all the HOWTOs about tuning Suricata for 10Gbit/sec. And each
> of them says something else. Meh. That's why I'm looking for your comments -
> I can't sleep, fighting battles in my brain ;)

Yes there are all bound to differ from one another - since the set up
is different for each  particular case. Example - some setups have 10
cores other 20 others 16 all with different CPU speed. Performance
depends also on the type of traffic , number of rules loaded, pps, avg
size of packets, NIC (Intel, Myricom,Tilera...).

So there is no one set up to rule them all :) . I think you should
test/experiment/adjust until you find which config combination works
best and is optimal for your scenario. (then keep an eye on the stats
on a regular basis :) )

>
> There are three possible scenarios here:
>
> 1. Leave HT enabled, don't touch affinity, leave scheduling to Linux
>
> In this setup Linux sometimes schedules workers on a "virtual" (HT) cores.
> And that is bad, because two workers compete for resources of the same
> physical core. Am I wrong here? I've seen Linux doing that.
> Also, cache coherency sucks here. L2 and L3 to the rescue, a bit. And
> migrating thread between cores should invalidate TLB (partially).
>
> 2. Disable HT, don't touch affinity, leave scheduling to Linux.
>
> Haven't tried it yet. It should help in theory.
>
> 3. Pin threads to physical cores.
>
> But, Suricata uses not just 16 threads for workers (in my setup). There are
> different "management/housekeeping" ones as well.
>
> Should I reserve some cores for them and set affinity that all of them can
> compete for like 1-2 cores, and then pin workers (less of them) to what's
> remaining?
>
> Or maybe pin 16 workers to cores and let the rest float as they wish?
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Training now available: http://suricata-ids.org/training/



-- 
Regards,
Peter Manev