<div dir="ltr">Hi Sean,<div><br></div><div>Looks like it helped some. Modified the cpu-set settings as you mentioned, and now loss is around 4-5% [capture.kernel_packets: 685173701, capture.kernel_drops: 8692212 ]</div><div><br></div><div>I will see if I can recompile Suricata with Hyper-Scan and see if the kernel_drops reduce to a lower number.</div><div><br></div><div>Thanks!</div><div>Fatema.</div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 12, 2018 at 9:12 AM, Cloherty, Sean E <span dir="ltr"><<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_2154441028046591105WordSection1">
<p class="MsoNormal">So looking at the docs – for runmode workers these are the two affinity settings which you need to concern yourself with – and the worker-cpu set is the critical one.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="background:rgb(238,255,204)"><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">management</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">cpu</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(0,112,32)">set</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">
</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)"> used
</span><b><span lang="EN" style="font-family:Consolas;color:rgb(0,112,32)">for</span></b><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)"> management (example
</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)"> flow</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">.</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">managers,
flow</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">.</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">recyclers)<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">worker</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">cpu</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(0,112,32)">set</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">
</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)">-</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)"> used
</span><b><span lang="EN" style="font-family:Consolas;color:rgb(0,112,32)">for</span></b><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)"> receive,streamtcp,decode,<wbr>detect,output(logging),respond</span><span lang="EN" style="font-family:Consolas;color:rgb(102,102,102)"><wbr>/</span><span lang="EN" style="font-family:Consolas;color:rgb(64,64,64)">reject</span><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">What you want to do is to use that list in node 0 as the ones to use for workers and then pick any two for the management CPU from node one –<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">So <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> cpu-affinity:<u></u><u></u></p>
<p class="MsoNormal"> - management-cpu-set:<u></u><u></u></p>
<p class="MsoNormal"> cpu: [ 37,39 ] # include only these cpus in affinity settings<u></u><u></u></p><span class="gmail-">
<p class="MsoNormal"> - receive-cpu-set:<u></u><u></u></p>
<p class="MsoNormal"> cpu: [ 0 ] # include only these cpus in affinity settings<u></u><u></u></p>
<p class="MsoNormal"> - worker-cpu-set:<u></u><u></u></p>
</span><p class="MsoNormal"> cpu: [ 4,6,8,10,12,14,16,18,20,22,24,<wbr>26,28,30,32,34,36,38 ]<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"> mode: "exclusive"<u></u><u></u></p>
<p class="MsoNormal"> # Use explicitly 3 threads and don't compute number by using<u></u><u></u></p>
<p class="MsoNormal"> # detect-thread-ratio variable:<u></u><u></u></p>
<p class="MsoNormal"> threads: 18<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span class="gmail-"><b>From:</b> fatema bannatwala [mailto:<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@<wbr>gmail.com</a>]
<br>
</span><b>Sent:</b> Wednesday, July 11, 2018 15:03 PM</p><div><div class="gmail-h5"><br>
<b>To:</b> Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>><br>
<b>Cc:</b> <a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">oisf-users@lists.<wbr>openinfosecfoundation.org</a><br>
<b>Subject:</b> Re: [Oisf-users] High Suricata capture.kernel_drops<u></u><u></u></div></div><p></p><div><div class="gmail-h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Sean.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have two NUMA nodes, and Node 0 is the NICs NUMA node:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,<wbr>22,24,26,28,30,32,34,36,38<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,<wbr>23,25,27,29,31,33,35,37,39<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">$ cat /sys/class/net/em1/device/<wbr>numa_node<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">0<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So does that mean that I can assign only threads from NUMA node0 to the management-cpu-set and worker-cpu-set, as it's the NICs NUMA node?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12pt;background:white">I am not able to figure out from Septun doc that what threads/cores would be pinned to which set in cpu-affinity, as you suggested earlier, hence went with "all" in worker and cpu sets by
default.</span> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I will try to update the drivers for the NICs next.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">As for HS, I didn't know about it before, and now that I have already compiled Suricata from source, and do $suricata --buil-info, if shows "Hyperscan support: no".<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Hence assuming that I have to recompile suricata again to get that enabled, which I would not like to do as of now.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Fatema.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Jul 11, 2018 at 2:19 PM, Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>> wrote:<u></u><u></u></p>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal">First get the NUMA node for the CPUs – lscpu should provide that in the last two lines of the output.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Find your NICs NUMA node 1<sup>st</sup> and go from there for affinity settings
<span style="font-size:10pt;font-family:Consolas">cat <span style="background:rgb(239,240,241)">
/sys/class/net/em1/device/<wbr>numa_node</span></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)">Update the drivers for the NIC -
<a href="https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=82947" target="_blank">
https://downloadcenter.intel.<wbr>com/download/24411/Intel-<wbr>Network-Adapter-Driver-for-<wbr>PCIe-40-Gigabit-Ethernet-<wbr>Network-Connections-Under-<wbr>Linux-?product=82947</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)">(Just remember that you will need to repeat this after any kernel updates)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:10pt;font-family:Consolas;background:rgb(239,240,241)"> </span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>From:</b> fatema bannatwala [mailto:<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@<wbr>gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, July 11, 2018 13:55 PM<br>
<b>To:</b> Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>><br>
<b>Cc:</b> <a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">
oisf-users@lists.<wbr>openinfosecfoundation.org</a><br>
<b>Subject:</b> Re: [Oisf-users] High Suricata capture.kernel_drops<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hi Sean,<u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for some quick points and recommendations.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I will work through those, and see if it helps.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">The documentation refers the tuning assuming two NICs p1p1 and p1p3, which was getting me confused, as I only have single NIC with 20 cores and 40 online threads, so was struggling
to set the config options right in the yaml file for cpu_affinity. I will try the hard coded method instead of all and see if it helps.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Fatema.<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div></div>