<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.n
{mso-style-name:n;}
span.o2
{mso-style-name:o2;
color:#666666;}
span.nb2
{mso-style-name:nb2;
color:#007020;}
span.k2
{mso-style-name:k2;
color:#007020;
font-weight:bold;}
span.p
{mso-style-name:p;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">So looking at the docs – for runmode workers these are the two affinity settings which you need to concern yourself with – and the worker-cpu set is the critical one.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="background:#EEFFCC"><span lang="EN" style="font-family:Consolas;color:#404040">management</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#404040">cpu</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#007020">set</span><span lang="EN" style="font-family:Consolas;color:#404040">
</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#404040"> used
</span><b><span lang="EN" style="font-family:Consolas;color:#007020">for</span></b><span lang="EN" style="font-family:Consolas;color:#404040"> management (example
</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#404040"> flow</span><span lang="EN" style="font-family:Consolas;color:#666666">.</span><span lang="EN" style="font-family:Consolas;color:#404040">managers,
flow</span><span lang="EN" style="font-family:Consolas;color:#666666">.</span><span lang="EN" style="font-family:Consolas;color:#404040">recyclers)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN" style="font-family:Consolas;color:#404040">worker</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#404040">cpu</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#007020">set</span><span lang="EN" style="font-family:Consolas;color:#404040">
</span><span lang="EN" style="font-family:Consolas;color:#666666">-</span><span lang="EN" style="font-family:Consolas;color:#404040"> used
</span><b><span lang="EN" style="font-family:Consolas;color:#007020">for</span></b><span lang="EN" style="font-family:Consolas;color:#404040"> receive,streamtcp,decode,detect,output(logging),respond</span><span lang="EN" style="font-family:Consolas;color:#666666">/</span><span lang="EN" style="font-family:Consolas;color:#404040">reject</span><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What you want to do is to use that list in node 0 as the ones to use for workers and then pick any two for the management CPU from node one –<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> cpu-affinity:<o:p></o:p></p>
<p class="MsoNormal"> - management-cpu-set:<o:p></o:p></p>
<p class="MsoNormal"> cpu: [ 37,39 ] # include only these cpus in affinity settings<o:p></o:p></p>
<p class="MsoNormal"> - receive-cpu-set:<o:p></o:p></p>
<p class="MsoNormal"> cpu: [ 0 ] # include only these cpus in affinity settings<o:p></o:p></p>
<p class="MsoNormal"> - worker-cpu-set:<o:p></o:p></p>
<p class="MsoNormal"> cpu: [ 4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38 ]<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> mode: "exclusive"<o:p></o:p></p>
<p class="MsoNormal"> # Use explicitly 3 threads and don't compute number by using<o:p></o:p></p>
<p class="MsoNormal"> # detect-thread-ratio variable:<o:p></o:p></p>
<p class="MsoNormal"> threads: 18<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><b>From:</b> fatema bannatwala [mailto:fatema.bannatwala@gmail.com]
<br>
<b>Sent:</b> Wednesday, July 11, 2018 15:03 PM<br>
<b>To:</b> Cloherty, Sean E <scloherty@mitre.org><br>
<b>Cc:</b> oisf-users@lists.openinfosecfoundation.org<br>
<b>Subject:</b> Re: [Oisf-users] High Suricata capture.kernel_drops<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Sean.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I have two NUMA nodes, and Node 0 is the NICs NUMA node:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">NUMA node0 CPU(s): 0,2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32,34,36,38<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">NUMA node1 CPU(s): 1,3,5,7,9,11,13,15,17,19,21,23,25,27,29,31,33,35,37,39<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">$ cat /sys/class/net/em1/device/numa_node<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">0<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So does that mean that I can assign only threads from NUMA node0 to the management-cpu-set and worker-cpu-set, as it's the NICs NUMA node?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:12.0pt;background:white">I am not able to figure out from Septun doc that what threads/cores would be pinned to which set in cpu-affinity, as you suggested earlier, hence went with "all" in worker and cpu sets by
default.</span> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I will try to update the drivers for the NICs next.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">As for HS, I didn't know about it before, and now that I have already compiled Suricata from source, and do $suricata --buil-info, if shows "Hyperscan support: no".<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Hence assuming that I have to recompile suricata again to get that enabled, which I would not like to do as of now.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Fatema.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Jul 11, 2018 at 2:19 PM, Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">First get the NUMA node for the CPUs – lscpu should provide that in the last two lines of the output.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Find your NICs NUMA node 1<sup>st</sup> and go from there for affinity settings
<span style="font-size:10.0pt;font-family:Consolas">cat <span style="background:#EFF0F1">
/sys/class/net/em1/device/numa_node</span></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1">Update the drivers for the NIC -
<a href="https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=82947" target="_blank">
https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=82947</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1">(Just remember that you will need to repeat this after any kernel updates)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:Consolas;background:#EFF0F1"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> fatema bannatwala [mailto:<a href="mailto:fatema.bannatwala@gmail.com" target="_blank">fatema.bannatwala@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, July 11, 2018 13:55 PM<br>
<b>To:</b> Cloherty, Sean E <<a href="mailto:scloherty@mitre.org" target="_blank">scloherty@mitre.org</a>><br>
<b>Cc:</b> <a href="mailto:oisf-users@lists.openinfosecfoundation.org" target="_blank">
oisf-users@lists.openinfosecfoundation.org</a><br>
<b>Subject:</b> Re: [Oisf-users] High Suricata capture.kernel_drops<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Sean,<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks for some quick points and recommendations.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I will work through those, and see if it helps.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">The documentation refers the tuning assuming two NICs p1p1 and p1p3, which was getting me confused, as I only have single NIC with 20 cores and 40 online threads, so was struggling
to set the config options right in the yaml file for cpu_affinity. I will try the hard coded method instead of all and see if it helps.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Fatema.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>