<div dir="ltr">Hello Experts,<div><br></div><div>Need some help tuning down our prod suricata box running Suricata v 5.0.2 with Myricom NIC: 10G-PCIE-8B-S myri_snf 3.0.20.50894</div><div><br></div><div>It is consistently reporting ~50% capture loss, calculated based off of the capture.kernel_packets and capture.kernel_dropped values reported in stats.log file.</div><div><br></div><div>I have followed the <a href="https://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/">https://blog.inliniac.net/2012/07/10/suricata-on-myricom-capture-cards/</a></div><div>guide to pin the cpus to the worker nodes and use pcap.buffer_size to increase the SNF dataring size, but no effect..</div><div><br></div><div>We have one Myri card connected to p2p1 and two NUMA nodes, each with 8 cores (16 HT):</div>NUMA node0 CPU(s): 0-7,16-23<br>NUMA node1 CPU(s): 8-15,24-31<div>OS: Centos 7</div><div><br></div><div>Any help in the right direction would be appreciated! :)</div><div><br></div><div>Thanks!</div><div>Fatema</div><div><br></div><div>Following is settings from suricata.yml file<br>
<div><br></div><div>
<p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"># Myricom support</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">pcap:</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>- interface: p2p1</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>threads: 14</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>buffer-size: 2gb</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>checksum-checks: no</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">pcap-file:</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">
</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>checksum-checks: auto</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">threading:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>set-cpu-affinity: yes</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>cpu-affinity:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>- management-cpu-set:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>cpu: [ "0" ]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>mode: "balanced"</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>prio:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>default: "low"</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>- worker-cpu-set:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>cpu: [ "1-7","9-15" ]</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>mode: "exclusive"</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>prio:</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><span class="gmail-Apple-converted-space"> </span>default: "high"</span></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"><br></span></p>Following is the currently recorded stats.log:</div><div>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Date: 6/19/2020 -- 10:55:36 (uptime: 0d, 04h 04m 10s)</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">Counter <span class="gmail-Apple-converted-space"> </span>| TM Name <span class="gmail-Apple-converted-space"> </span>| Value</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">------------------------------------------------------------------------------------</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">capture.kernel_packets<span class="gmail-Apple-converted-space"> </span>| Total <span class="gmail-Apple-converted-space"> </span>| 28447139411</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">capture.kernel_drops<span class="gmail-Apple-converted-space"> </span>| Total <span class="gmail-Apple-converted-space"> </span>| 27910518132</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">capture.kernel_ifdrops<span class="gmail-Apple-converted-space"> </span>| Total <span class="gmail-Apple-converted-space"> </span>| 6034</span></p>
<p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">decoder.pkts<span class="gmail-Apple-converted-space"> </span>| Total <span class="gmail-Apple-converted-space"> </span>| 536633135</span></p></div><div><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><br></p><div><br></div>SNF parameters:</div><div><br><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><br></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SNF_APP_ID=32</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SNF_DATARING_SIZE=4096MB</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SNF_DESCRING_SIZE=1024MB</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SNF_NUM_RINGS=14</span></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">SNF_FLAGS=0x1</span></p><p class="gmail-p2" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255);min-height:18px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p><p class="gmail-p1" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">LD_PRELOAD="/opt/snf/lib/libpcap.so.1"</span></p><p class="gmail-p2" style="margin:0px;font:15px Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255);min-height:18px"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures"></span><br></p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)">
</p><p class="gmail-p1" style="margin:0px;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:15px;line-height:normal;font-family:Menlo;color:rgb(101,13,108);background-color:rgb(252,248,255)"><span class="gmail-s1" style="font-variant-ligatures:no-common-ligatures">OPTIONS="--user suricata --group suricata --pcap"</span></p></div></div></div>