<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Franklin Gothic Book";
panose-1:2 11 5 3 2 1 2 2 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.m-2935738683436375521gmail-
{mso-style-name:m_-2935738683436375521gmail-;}
span.m-2935738683436375521gmail-hoenzb
{mso-style-name:m_-2935738683436375521gmail-hoenzb;}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi Alexander, Erich,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have noticed significant bursts of drops on our Myricom sensors as well. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I am wondering if you’ve noticed this as well: During periods of drops, I notice one of the workers is pegged and the others are almost idle. I have bypass enabled too. I am curious if this is an “elephant flow” – a disproportionate amount of traffic being hashed to a single worker via the Myricom RSS. I’m not sure. It eventually seems to resolve itself, but not after significant loss. The strange part to me is how low the usage on the other works is when this happens.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>My yaml settings are not quite as generous as Erichs, but I will try tweaking them. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-family:"Lucida Console"'>$ top -p $(pgrep Suricata-Main) -H</span><o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console";color:black;background:silver;mso-highlight:silver'> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><b><span style='font-family:"Lucida Console"'>28092 suri+ 20 0 0.173t 0.172t 0.156t R 99.7 70.2 6125:10 W#02-snf0<o:p></o:p></span></b></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28107 suri+ 20 0 0.173t 0.172t 0.156t S 1.3 70.2 731:43.64 FM#01<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28091 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3217:08 W#01-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28095 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3369:32 W#05-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28096 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3401:27 W#06-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28097 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 6416:49 W#07-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28100 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 2837:30 W#10-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28101 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 7366:56 W#11-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28102 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 2792:57 W#12-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28103 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3203:58 W#13-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28105 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 3508:24 W#15-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28106 suri+ 20 0 0.173t 0.172t 0.156t S 0.7 70.2 6806:13 W#16-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28093 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 3353:10 W#03-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28094 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 4036:36 W#04-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28098 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 2819:50 W#08-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28099 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 3948:24 W#09-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28104 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 3068:59 W#14-snf0<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28108 suri+ 20 0 0.173t 0.172t 0.156t S 0.3 70.2 339:50.10 FR#01<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28071 suri+ 20 0 0.173t 0.172t 0.156t S 0.0 70.2 17:07.49 Suricata-Main<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28109 suri+ 20 0 0.173t 0.172t 0.156t S 0.0 70.2 0:10.36 CW<o:p></o:p></span></p><p class=MsoNormal style='text-autospace:none'><span style='font-family:"Lucida Console"'>28110 suri+ 20 0 0.173t 0.172t 0.156t S 0.0 70.2 0:25.10 CS<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>One other note, if you’d like to see the debug info, try setting the SNF_DEBUG_FILENAME to a path (i.e. <span style='font-family:"Lucida Console"'>SNF_DEBUG_FILENAME='/tmp/snf.out'</span> ) in additional to the SNF_DEBUG_MASK. That seems to work for me.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>Zach<o:p></o:p></p><p class=MsoNormal><b><span style='font-size:12.0pt;font-family:"Franklin Gothic Book",sans-serif'>________________________<o:p></o:p></span></b></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif'>Zach Rasmor<o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Franklin Gothic Book",sans-serif'>Email: <a href="mailto:zachary.r.rasmor@lmco.com">zachary.r.rasmor@lmco.com</a></span><b><span style='font-size:10.0pt;font-family:"Franklin Gothic Book",sans-serif'><o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Franklin Gothic Book",sans-serif'>Office: 301.921.7080<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><b>From:</b> Oisf-users [mailto:oisf-users-bounces@lists.openinfosecfoundation.org] <b>On Behalf Of </b>Erich Lerch<br><b>Sent:</b> Wednesday, February 21, 2018 2:45 AM<br><b>To:</b> oisf-users@lists.openinfosecfoundation.org<br><b>Subject:</b> EXTERNAL: [Oisf-users] Fwd: Installing / Running Suricata with Myricom NICs<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hi Alex<o:p></o:p></p><div><div><div><div><div><div><div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>We seem to have a similar setup as of OS, hardware, traffic and myricom.<o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'>We experience almost no drops (usually less than 0.2%).<o:p></o:p></p></div><div><p class=MsoNormal>We start suri with these params:<br><br>SNF_NUM_RINGS=10 SNF_FLAGS=0x1 SNF_DATARING_SIZE=12884901888 SNF_DESCRING_SIZE=3221225472<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>We do start Suri with "-i snf0<span class=m-2935738683436375521gmail->", it throws a warning in suricata.log, but it works:</span><br><span class=m-2935738683436375521gmail-><Warning> - [ERRCODE: SC_ERR_SYSCALL(50)] - Failure when trying to get feature via ioctl for 'snf0': No such device (19)</span><o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><o:p> </o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'>There are other factors which influence the performance:<o:p></o:p></p></div><p class=MsoNormal style='margin-bottom:12.0pt'>- I set a BPF to bypass traffic I don't want to see anyway<o:p></o:p></p></div><div><p class=MsoNormal>- number of rules (we have activated about 22'000 rules)<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal>- suricata.yaml, some configs greatly influence performance<o:p></o:p></p></div><div><p class=MsoNormal>Some excerpts:<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>...<br>pcap:<br> - interface: snf0<br> threads: 10 <<--- has to correspond with SNF_NUM_RINGS<br> buffer-size: 2gb<br> checksum-checks: no<br> promisc: no<br> snaplen: 1520<br> bpf-filter: "..."<br><br>...<br>stream:<br> memcap: 5gb<br>...<br> reassembly:<br> memcap: 10gb<br>...<br><br>detect:<br> profile: custom<br> custom-values:<br> toclient-groups: 200<br> toserver-groups: 200<br>...<br># hyperscan<br>mpm-algo: hs<br>spm-algo: hs<o:p></o:p></p></div><p class=MsoNormal># pin to cores<o:p></o:p></p><div><p class=MsoNormal style='margin-bottom:12.0pt'>threading:<br> set-cpu-affinity: yes<br> cpu-affinity:<br> - management-cpu-set:<br> cpu: [ ... ]<br> - worker-cpu-set:<br> cpu: [ ... ]<br> mode: "exclusive" # run detect threads in these cpus<br> threads: 10<br> prio:<br> high: [ ... ]<br> default: "medium"<br>...<br>max-pending-packets: 8192<br>...<br>flow:<br> memcap: 1024mb<br> hash-size: 524288<br> prealloc: 1048576<br> emergency-recovery: 30<br> managers: 2<br> recyclers: 2<br>...<br><br><o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>- try to pin suri worker threads to the same NUMA node the myricom is attached to<br><br><o:p></o:p></p></div><div><p class=MsoNormal>HTH,<o:p></o:p></p></div><div><p class=MsoNormal>erich<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal><o:p> </o:p></p><div><div><div><p class=MsoNormal>2018-02-20 18:58 GMT+01:00 Alexander Merck <<a href="mailto:alexander.merck@duke.edu" target="_blank">alexander.merck@duke.edu</a>>:<o:p></o:p></p></div></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in'><div><div><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hello,<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Hopefully someone can help shed some light on some issues we've been seeing. We just installed a new instance of Suricata on a fresh RHEL7 monitoring box with Myricom cards. However, we are seeing significant packet loss (20-35%) on 2-3 Gbps traffic when attempting to use the SNF drivers.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>I'm suspecting that the Myricom SNF drivers are not functioning as expected. We're able to run tcpdump compiled against these drivers with no issue, including generating debug output. We've also found when supplying the SNF_DEBUG_MASK environment variable when running Suricata, no debug output is generated.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Also, when using Suricata with the SNF drivers, should you be able to use the interface names specified by SNF (e.g. snf0)? When trying to run Suricata using the -i snf0, we get an "Unable to find iface snf0: No such device" error message. We are only able to run Suricata against the interface names specified by the kernel (in our case, enp4s0)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The version of Suricata is 4.0.4 and the version of SNF is 3.0.12. Running ldd shows that Suricata is linked against the SNF libraries.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'># ldd /usr/bin/suricata<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>...<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> libpcap.so.1 => /opt/snf/lib/libpcap.so.1 (0x00007f238ffb0000)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>...<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> libsnf.so.0 => /opt/snf/lib/libsnf.so.0 (0x00007f238dae4000)<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>We compiled Suricata per these instructions: <a href="https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom" target="_blank">https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Myricom</a>. I did notice that this document is over five years old, but all of the configuration options seemed correct.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>./configure --with-libpcap-includes=/opt/snf/include/ --with-libpcap-libraries=/opt/snf/lib/ --prefix=/usr --sysconfdir=/etc --localstatedir=/var<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>And we're running Suricata with the following command:<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>SNF_NUM_RINGS=32 SNF_DATARING_SIZE=17179869184 SNF_DESCRING_SIZE=4294967296 SNF_FLAGS=0x1 SNF_DEBUG_MASK=3 SNF_DEBUG_FILENAME="/tmp/snf.out" /usr/bin/suricata -c /etc/suricata/suricata.yaml -i enp4s0 --runmode=workers<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>The box we’re running this on has 64 cores and 256GB of RAM, so I doubt it’s a resource issue…but could potentially be a configuration issue.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>Are we missing something in the install process that may be causing these issues? Any recommendations or pointers would be greatly appreciated. Thanks!<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'>-Alex M<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#888888'> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='color:#888888'> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.5pt;color:black'>-- </span><span style='color:#888888'><o:p></o:p></span></p><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.5pt;color:black'>Alexander Merck</span><span style='color:#888888'><o:p></o:p></span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.5pt;color:black'>Duke University</span><span style='color:#888888'><o:p></o:p></span></p></div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span style='font-size:10.5pt;color:black'>IT Security Office</span><span style='color:#888888'><o:p></o:p></span></p></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div><p class=MsoNormal>_______________________________________________<br>Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org" target="_blank">oisf-users@openinfosecfoundation.org</a><br>Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br><br>Conference: <a href="https://suricon.net" target="_blank">https://suricon.net</a><br>Trainings: <a href="https://suricata-ids.org/training/" target="_blank">https://suricata-ids.org/training/</a><o:p></o:p></p></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></div><p class=MsoNormal><o:p> </o:p></p></div></div></body></html>