<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Thanks Eric and Coop. I’ve had a death in the family and won’t be able to work on this for a few more days, but I’ll try the cluster_flow setting (though in my previous attempts that seemed to result in quite a high % of kernel_drops) and also the trick Coop talked about with setting the cores to even/odd irq balancing. I like that last idea, but will be a little tricky.<div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 13, 2015, at 11:50 AM, Eric Leblond <<a href="mailto:eric@regit.org" class="">eric@regit.org</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Hello,</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On Fri, 2015-02-13 at 11:45 -0600, Barkley, Joey wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">Yes, sort of…<br class=""><br class="">I have 64 cores available. I set threads: 32 and cpu: [ 0-31].<br class=""><br class="">It creates 32 threads, but only 16 of them show up as actually processing data in the stats log file. cores 15-31 always show 0 for kernel_packets and kernel_drops. Does that mean it just doesn’t need the extra cores? I do have some (very minimal) drops, but I’d think that if I had anything more than 0 it would start using more cores for processing.<br class=""><br class="">And if it matters, this is 2.1beta3.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Your configuration is based on the RSS load balancing done by the</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">network card. And most cards (like the ixgbe) are limited to 16 queues.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">The consequence is that only 16 threads will receive packets even if you</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">ask for more due to the limitation of the card.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">One possible try is to set cluster_flow in the af-packet configuration.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Not sure it will work but it could worth the try. In that case, the</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">kernel is doing the load balancing and he is no such limitation.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">++</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br class=""><br class=""><blockquote type="cite" class="">On Feb 13, 2015, at 11:36 AM, Cooper F. Nelson <<a href="mailto:cnelson@ucsd.edu" class="">cnelson@ucsd.edu</a>> wrote:<br class=""><br class="">-----BEGIN PGP SIGNED MESSAGE-----<br class="">Hash: SHA1<br class=""><br class="">The number of threads is governed by this configuration:<br class=""><br class=""><blockquote type="cite" class="">af-packet:<br class="">- interface: eth2<br class=""> # Number of receive threads (>1 will enable experimental flow pinned<br class=""> # runmode)<br class=""> threads: 16<br class=""></blockquote><br class="">... and this:<br class=""><br class=""><blockquote type="cite" class=""> - detect-cpu-set:<br class=""> cpu: [ 0-15 ]<br class=""> mode: "exclusive" # run detect threads in these cpus<br class=""> # Use explicitely 3 threads and don't compute number by using<br class=""> # detect-thread-ratio variable:<br class=""> #threads: 2<br class=""> prio:<br class=""> default: "high"<br class=""></blockquote><br class="">Are they both set to use all available cores?<br class=""><br class="">- -Coop<br class=""><br class="">On 2/13/2015 8:47 AM, Barkley, Joey wrote:<br class=""><blockquote type="cite" class="">All,<span class="Apple-converted-space"> </span><br class=""><br class="">I have made significant progress in tuning our suricata instance to<br class="">handle our network traffic. Thanks to everyone who has helped me.<br class=""><br class="">Question: Regardless of how many threads I configure, suricata only<br class="">shows kernel_packets and kernel_drops for the first 16 threads. Is there<br class="">a hard limit of 16 “usable” threads? My system has 64 cores but it<br class="">doesn’t seem like I’m able to use more than 16 cores. Have I just<br class="">configured something incorrectly? I have primarily followed advice on<br class="">this list and also on<br class=""><a href="http://pevma.blogspot.se/2013/12/suricata-and-grand-slam-of-open-source_8.html" class="">http://pevma.blogspot.se/2013/12/suricata-and-grand-slam-of-open-source_8.html</a> for<br class="">AF_PACKET configuration. Would it help for me to assign my<br class="">management-cpu-set to different cores than my detect-cpu-set? I seem to<br class="">remember reading that would not be good as it would adversely impact<br class="">performance. Or possibly, would increasing the detect-thread-ratio work?<br class="">I’m using cluster_cpu and not sure how that would be affected by<br class="">changing those settings.<br class=""><br class="">Advice welcome.<br class=""><br class="">Thanks,<br class="">Joey<br class=""><br class=""><br class="">_______________________________________________<br class="">Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" class="">oisf-users@openinfosecfoundation.org</a><br class="">Site: <a href="http://suricata-ids.org" class="">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" class="">http://suricata-ids.org/support/</a><br class="">List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" class="">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br class="">Training now available: <a href="http://suricata-ids.org/training/" class="">http://suricata-ids.org/training/</a><br class=""><br class=""></blockquote><br class=""><br class="">- --<span class="Apple-converted-space"> </span><br class="">Cooper Nelson<br class="">Network Security Analyst<br class="">UCSD ACT Security Team<br class=""><a href="mailto:cnelson@ucsd.edu" class="">cnelson@ucsd.edu</a> x41042<br class="">-----BEGIN PGP SIGNATURE-----<br class="">Version: GnuPG v2.0.17 (MingW32)<br class=""><br class="">iQEcBAEBAgAGBQJU3jY0AAoJEKIFRYQsa8FWpVEH/1U6bQmS5eJwUtlR59KQL8lh<br class="">R0N5rhexMvMEKD9ES5NhfHw3lEymPTK7Z/1AF797A7TA3d5vcRYgnt7n9pFiJB9L<br class="">QuUjcHxochZcfujUbFOWmMvC4EtqQtSbY/zVCrZgUJW8nkfAiMNSPAKJwY+YO+W3<br class="">LL4CfdVbQTGEb7eWLXH3wjnVDXvQXmoqOlI+QTR3SKrxksBk+54169pZPme7Vivj<br class="">GCidKOiGtsTFU0UGY4geAlhw3WABa3tz4m8oYBEoLOAXnua+uOlxd2zgDy7MeU5+<br class="">aKf/sw4eEBdW9nAnunHN+u/TeE9bvlai7K5WtF0il2p3F2y4841fP7gpeIDlSGQ=<br class="">=IilM<br class="">-----END PGP SIGNATURE-----<br class=""><br class=""></blockquote><br class=""><br class="">_______________________________________________<br class="">Suricata IDS Users mailing list:<span class="Apple-converted-space"> </span><a href="mailto:oisf-users@openinfosecfoundation.org" class="">oisf-users@openinfosecfoundation.org</a><br class="">Site:<span class="Apple-converted-space"> </span><a href="http://suricata-ids.org/" class="">http://suricata-ids.org</a><span class="Apple-converted-space"> </span>| Support:<span class="Apple-converted-space"> </span><a href="http://suricata-ids.org/support/" class="">http://suricata-ids.org/support/</a><br class="">List:<span class="Apple-converted-space"> </span><a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" class="">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br class="">Training now available:<span class="Apple-converted-space"> </span><a href="http://suricata-ids.org/training/" class="">http://suricata-ids.org/training/</a><br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">--<span class="Apple-converted-space"> </span></span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Eric Leblond <</span><a href="mailto:eric@regit.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class="">eric@regit.org</a><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">></span></div></blockquote></div><br class=""></div></body></html>