[Oisf-users] Processing threads limit of 16?

[Eric Leblond]

Hello,

On Fri, 2015-02-13 at 11:45 -0600, Barkley, Joey wrote:
> Yes, sort of…
> 
> I have 64 cores available. I set threads: 32 and cpu: [ 0-31].
> 
> It creates 32 threads, but only 16 of them show up as actually processing data in the stats log file. cores 15-31 always show 0 for kernel_packets and kernel_drops. Does that mean it just doesn’t need the extra cores? I do have some (very minimal) drops, but I’d think that if I had anything more than 0 it would start using more cores for processing.
> 
> And if it matters, this is 2.1beta3.

Your configuration is based on the RSS load balancing done by the
network card. And most cards (like the ixgbe) are limited to 16 queues.
The consequence is that only 16 threads will receive packets even if you
ask for more due to the limitation of the card.

One possible try is to set cluster_flow in the af-packet configuration.
Not sure it will work but it could worth the try. In that case, the
kernel is doing the load balancing and he is no such limitation.

++

> 
> 
> > On Feb 13, 2015, at 11:36 AM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:
> > 
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> > 
> > The number of threads is governed by this configuration:
> > 
> >> af-packet:
> >>  - interface: eth2
> >>    # Number of receive threads (>1 will enable experimental flow pinned
> >>    # runmode)
> >>    threads: 16
> > 
> > ... and this:
> > 
> >>    - detect-cpu-set:
> >>        cpu: [ 0-15 ]
> >>        mode: "exclusive" # run detect threads in these cpus
> >>        # Use explicitely 3 threads and don't compute number by using
> >>        # detect-thread-ratio variable:
> >>        #threads: 2
> >>        prio:
> >>          default: "high"
> > 
> > Are they both set to use all available cores?
> > 
> > - -Coop
> > 
> > On 2/13/2015 8:47 AM, Barkley, Joey wrote:
> >> All, 
> >> 
> >> I have made significant progress in tuning our suricata instance to
> >> handle our network traffic. Thanks to everyone who has helped me.
> >> 
> >> Question: Regardless of how many threads I configure, suricata only
> >> shows kernel_packets and kernel_drops for the first 16 threads. Is there
> >> a hard limit of 16 “usable” threads? My system has 64 cores but it
> >> doesn’t seem like I’m able to use more than 16 cores. Have I just
> >> configured something incorrectly? I have primarily followed advice on
> >> this list and also on
> >> http://pevma.blogspot.se/2013/12/suricata-and-grand-slam-of-open-source_8.html for
> >> AF_PACKET configuration. Would it help for me to assign my
> >> management-cpu-set to different cores than my detect-cpu-set? I seem to
> >> remember reading that would not be good as it would adversely impact
> >> performance. Or possibly, would increasing the detect-thread-ratio work?
> >> I’m using cluster_cpu and not sure how that would be affected by
> >> changing those settings.
> >> 
> >> Advice welcome.
> >> 
> >> Thanks,
> >> Joey
> >> 
> >> 
> >> _______________________________________________
> >> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> >> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >> Training now available: http://suricata-ids.org/training/
> >> 
> > 
> > 
> > - -- 
> > Cooper Nelson
> > Network Security Analyst
> > UCSD ACT Security Team
> > cnelson at ucsd.edu x41042
> > -----BEGIN PGP SIGNATURE-----
> > Version: GnuPG v2.0.17 (MingW32)
> > 
> > iQEcBAEBAgAGBQJU3jY0AAoJEKIFRYQsa8FWpVEH/1U6bQmS5eJwUtlR59KQL8lh
> > R0N5rhexMvMEKD9ES5NhfHw3lEymPTK7Z/1AF797A7TA3d5vcRYgnt7n9pFiJB9L
> > QuUjcHxochZcfujUbFOWmMvC4EtqQtSbY/zVCrZgUJW8nkfAiMNSPAKJwY+YO+W3
> > LL4CfdVbQTGEb7eWLXH3wjnVDXvQXmoqOlI+QTR3SKrxksBk+54169pZPme7Vivj
> > GCidKOiGtsTFU0UGY4geAlhw3WABa3tz4m8oYBEoLOAXnua+uOlxd2zgDy7MeU5+
> > aKf/sw4eEBdW9nAnunHN+u/TeE9bvlai7K5WtF0il2p3F2y4841fP7gpeIDlSGQ=
> > =IilM
> > -----END PGP SIGNATURE-----
> > 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Training now available: http://suricata-ids.org/training/

-- 
Eric Leblond <eric at regit.org>