[Oisf-users] Processing threads limit of 16?

[Barkley, Joey]

Thanks Eric and Coop. I’ve had a death in the family and won’t be able to work on this for a few more days, but I’ll try the cluster_flow setting (though in my previous attempts that seemed to result in quite a high % of kernel_drops) and also the trick Coop talked about with setting the cores to even/odd irq balancing. I like that last idea, but will be a little tricky.


On Feb 13, 2015, at 11:50 AM, Eric Leblond <eric at regit.org<mailto:eric at regit.org>> wrote:

Hello,

On Fri, 2015-02-13 at 11:45 -0600, Barkley, Joey wrote:
Yes, sort of…

I have 64 cores available. I set threads: 32 and cpu: [ 0-31].

It creates 32 threads, but only 16 of them show up as actually processing data in the stats log file. cores 15-31 always show 0 for kernel_packets and kernel_drops. Does that mean it just doesn’t need the extra cores? I do have some (very minimal) drops, but I’d think that if I had anything more than 0 it would start using more cores for processing.

And if it matters, this is 2.1beta3.

Your configuration is based on the RSS load balancing done by the
network card. And most cards (like the ixgbe) are limited to 16 queues.
The consequence is that only 16 threads will receive packets even if you
ask for more due to the limitation of the card.

One possible try is to set cluster_flow in the af-packet configuration.
Not sure it will work but it could worth the try. In that case, the
kernel is doing the load balancing and he is no such limitation.

++



On Feb 13, 2015, at 11:36 AM, Cooper F. Nelson <cnelson at ucsd.edu<mailto:cnelson at ucsd.edu>> wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

The number of threads is governed by this configuration:

af-packet:
- interface: eth2
  # Number of receive threads (>1 will enable experimental flow pinned
  # runmode)
  threads: 16

... and this:

  - detect-cpu-set:
      cpu: [ 0-15 ]
      mode: "exclusive" # run detect threads in these cpus
      # Use explicitely 3 threads and don't compute number by using
      # detect-thread-ratio variable:
      #threads: 2
      prio:
        default: "high"

Are they both set to use all available cores?

- -Coop

On 2/13/2015 8:47 AM, Barkley, Joey wrote:
All,

I have made significant progress in tuning our suricata instance to
handle our network traffic. Thanks to everyone who has helped me.

Question: Regardless of how many threads I configure, suricata only
shows kernel_packets and kernel_drops for the first 16 threads. Is there
a hard limit of 16 “usable” threads? My system has 64 cores but it
doesn’t seem like I’m able to use more than 16 cores. Have I just
configured something incorrectly? I have primarily followed advice on
this list and also on
http://pevma.blogspot.se/2013/12/suricata-and-grand-slam-of-open-source_8.html for
AF_PACKET configuration. Would it help for me to assign my
management-cpu-set to different cores than my detect-cpu-set? I seem to
remember reading that would not be good as it would adversely impact
performance. Or possibly, would increasing the detect-thread-ratio work?
I’m using cluster_cpu and not sure how that would be affected by
changing those settings.

Advice welcome.

Thanks,
Joey


_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org<mailto:oisf-users at openinfosecfoundation.org>
Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Training now available: http://suricata-ids.org/training/



- --
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu<mailto:cnelson at ucsd.edu> x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJU3jY0AAoJEKIFRYQsa8FWpVEH/1U6bQmS5eJwUtlR59KQL8lh
R0N5rhexMvMEKD9ES5NhfHw3lEymPTK7Z/1AF797A7TA3d5vcRYgnt7n9pFiJB9L
QuUjcHxochZcfujUbFOWmMvC4EtqQtSbY/zVCrZgUJW8nkfAiMNSPAKJwY+YO+W3
LL4CfdVbQTGEb7eWLXH3wjnVDXvQXmoqOlI+QTR3SKrxksBk+54169pZPme7Vivj
GCidKOiGtsTFU0UGY4geAlhw3WABa3tz4m8oYBEoLOAXnua+uOlxd2zgDy7MeU5+
aKf/sw4eEBdW9nAnunHN+u/TeE9bvlai7K5WtF0il2p3F2y4841fP7gpeIDlSGQ=
=IilM
-----END PGP SIGNATURE-----



_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org<mailto:oisf-users at openinfosecfoundation.org>
Site: http://suricata-ids.org<http://suricata-ids.org/> | Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Training now available: http://suricata-ids.org/training/

--
Eric Leblond <eric at regit.org<mailto:eric at regit.org>>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20150216/f615d27d/attachment-0002.html>