<div dir="ltr">Hi Mike,<div><br></div><div>As of now following is from the stats.log file:</div><div><br></div><div><div>------------------------------------------------------------------------------------</div><div>Date: 7/12/2018 -- 10:36:07 (uptime: 0d, 00h 42m 00s)</div><div>------------------------------------------------------------------------------------</div><div>Counter | TM Name | Value</div><div>------------------------------------------------------------------------------------</div><div>capture.kernel_packets | Total | 799263310</div><div>capture.kernel_drops | Total | 9026058</div><div>decoder.pkts | Total | 790263391</div><div>decoder.bytes | Total | 676508317106</div><div>decoder.invalid | Total | 1195</div><div>decoder.ipv4 | Total | 790263395</div><div>decoder.ipv6 | Total | 48047</div><div>decoder.ethernet | Total | 790263391</div><div>decoder.tcp | Total | 628832359</div><div>decoder.udp | Total | 123066710</div><div>decoder.icmpv4 | Total | 1166436</div><div>decoder.icmpv6 | Total | 30616</div><div>decoder.gre | Total | 4</div><div>decoder.teredo | Total | 15176</div><div>decoder.avg_pkt_size | Total | 856</div><div>decoder.max_pkt_size | Total | 15604</div><div>flow.tcp | Total | 8928156</div><div>flow.udp | Total | 2513270</div><div>flow.icmpv6 | Total | 1898</div><div>decoder.icmpv4.ipv4_unknown_ver | Total | 13</div><div>decoder.tcp.hlen_too_small | Total | 907</div><div>decoder.tcp.opt_invalid_len | Total | 272</div><div>decoder.udp.pkt_too_small | Total | 3</div><div>tcp.sessions | Total | 7476848</div><div>tcp.ssn_memcap_drop | Total | 755957</div><div>tcp.pseudo | Total | 146</div><div>tcp.invalid_checksum | Total | 25868</div><div>tcp.syn | Total | 9252585</div><div>tcp.synack | Total | 3002962</div><div>tcp.rst | Total | 2109777</div><div>tcp.segment_memcap_drop | Total | 239663</div><div>tcp.stream_depth_reached | Total | 24785</div><div>tcp.reassembly_gap | Total | 169111</div><div>tcp.overlap | Total | 520434</div><div>tcp.insert_data_normal_fail | Total | 81175273</div><div>tcp.insert_data_overlap_fail | Total | 78</div><div>detect.alert | Total | 10020</div><div>detect.mpm_list | Total | 4</div><div>detect.nonmpm_list | Total | 2</div><div>detect.fnonmpm_list | Total | 1</div><div>detect.match_list | Total | 4</div><div>app_layer.flow.http | Total | 4918</div><div>app_layer.tx.http | Total | 16709</div><div>app_layer.flow.ftp | Total | 1</div><div>app_layer.flow.smtp | Total | 54</div><div>app_layer.tx.smtp | Total | 105</div><div>app_layer.flow.tls | Total | 26584</div><div>app_layer.flow.ssh | Total | 11</div><div>app_layer.flow.dns_tcp | Total | 229</div><div>app_layer.tx.dns_tcp | Total | 212</div><div>app_layer.flow.failed_tcp | Total | 13012</div><div>app_layer.flow.dcerpc_udp | Total | 54</div><div>app_layer.flow.dns_udp | Total | 1446949</div><div>app_layer.tx.dns_udp | Total | 177476</div><div>app_layer.flow.failed_udp | Total | 1066267</div><div>flow_mgr.closed_pruned | Total | 2052437</div><div>flow_mgr.new_pruned | Total | 7080586</div><div>flow_mgr.est_pruned | Total | 1831390</div><div>flow.spare | Total | 10691</div><div>flow.emerg_mode_entered | Total | 71</div><div>flow.emerg_mode_over | Total | 71</div><div>flow.tcp_reuse | Total | 22144</div><div>flow_mgr.flows_checked | Total | 88100</div><div>flow_mgr.flows_notimeout | Total | 85619</div><div>flow_mgr.flows_timeout | Total | 2481</div><div>flow_mgr.flows_timeout_inuse | Total | 35</div><div>flow_mgr.flows_removed | Total | 2446</div><div>flow_mgr.rows_checked | Total | 65536</div><div>flow_mgr.rows_skipped | Total | 53175</div><div>flow_mgr.rows_empty | Total | 4</div><div>flow_mgr.rows_maxlen | Total | 20</div><div>tcp.memuse | Total | 66631640</div><div>tcp.reassembly_memuse | Total | 268435376</div><div>dns.memuse | Total | 17021644</div><div>dns.memcap_global | Total | 6500069</div><div>http.memuse | Total | 352527</div><div>flow.memuse | Total | 121777504</div></div><div><br></div><div><br></div><div>Thanks,</div><div>Fatema.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 12, 2018 at 3:20 AM, Michał Purzyński <span dir="ltr"><<a href="mailto:michalpurzynski1@gmail.com" target="_blank">michalpurzynski1@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Share your Suricata stats please. Nothing will work correctly when memory isn’t allocated correctly. Also part of Septun ;)<br>
<div class="HOEnZb"><div class="h5"><br>
> On Jul 11, 2018, at 11:43 PM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>> wrote:<br>
> <br>
>> On Thu, Jul 12, 2018 at 9:06 AM, Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>> wrote:<br>
>> <br>
>> <br>
>> On 11 Jul 2018, at 22:02, fatema bannatwala <<a href="mailto:fatema.bannatwala@gmail.com">fatema.bannatwala@gmail.com</a>><br>
>> wrote:<br>
>> <br>
>> Hi Sean.<br>
>> <br>
>> I have two NUMA nodes, and Node 0 is the NICs NUMA node:<br>
>> <br>
>> NUMA node0 CPU(s):<br>
>> 0,2,4,6,8,10,12,14,16,18,20,<wbr>22,24,26,28,30,32,34,36,38<br>
>> NUMA node1 CPU(s):<br>
>> 1,3,5,7,9,11,13,15,17,19,21,<wbr>23,25,27,29,31,33,35,37,39<br>
>> <br>
>> $ cat /sys/class/net/em1/device/<wbr>numa_node<br>
>> 0<br>
>> <br>
>> So does that mean that I can assign only threads from NUMA node0 to the<br>
>> management-cpu-set and worker-cpu-set, as it's the NICs NUMA node?<br>
>> <br>
>> <br>
>> <br>
>> There are two ways you can go by here (the way I see it) but I think the<br>
>> easiest from administrative point (to at least try out fast) might be to<br>
>> just use numactl (including membind if needed) to make sure Suri is using<br>
>> the NICs local NUMA<br>
>> <br>
>> I am not able to figure out from Septun doc that what threads/cores would be<br>
>> pinned to which set in cpu-affinity, as you suggested earlier, hence went<br>
>> with "all" in worker and cpu sets by default.<br>
>> <br>
>> I will try to update the drivers for the NICs next.<br>
>> <br>
>> <br>
>> That is always recommended !<br>
>> <br>
>> As for HS, I didn't know about it before, and now that I have already<br>
>> compiled Suricata from source, and do $suricata --buil-info, if shows<br>
>> "Hyperscan support: no".<br>
>> Hence assuming that I have to recompile suricata again to get that enabled,<br>
>> which I would not like to do as of now.<br>
>> <br>
>> <br>
>> There is an example here of how to compile Hyperscan on Ubuntu from the<br>
>> docs-<br>
>> <a href="https://suricata.readthedocs.io/en/latest/performance/hyperscan.html?highlight=Hyperscan" rel="noreferrer" target="_blank">https://suricata.readthedocs.<wbr>io/en/latest/performance/<wbr>hyperscan.html?highlight=<wbr>Hyperscan</a><br>
>> <br>
>> Thanks<br>
>> <br>
> <br>
> <br>
> Since we are on the subject - this example should get you the latest<br>
> Suricata with hyperscan (you may want to update the boost version<br>
> though ) on RedHat/CentOS-<br>
> <a href="https://pastebin.com/iSKK53Dw" rel="noreferrer" target="_blank">https://pastebin.com/iSKK53Dw</a><br>
> <br>
> Hope it helps!<br>
> <br>
>> <br>
>> Thanks,<br>
>> Fatema.<br>
>> <br>
>> <br>
>> <br>
>> <br>
>> On Wed, Jul 11, 2018 at 2:19 PM, Cloherty, Sean E <<a href="mailto:scloherty@mitre.org">scloherty@mitre.org</a>><br>
>> wrote:<br>
>>> <br>
>>> First get the NUMA node for the CPUs – lscpu should provide that in the<br>
>>> last two lines of the output.<br>
>>> <br>
>>> <br>
>>> <br>
>>> Find your NICs NUMA node 1st and go from there for affinity settings cat<br>
>>> /sys/class/net/em1/device/<wbr>numa_node<br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> Update the drivers for the NIC -<br>
>>> <a href="https://downloadcenter.intel.com/download/24411/Intel-Network-Adapter-Driver-for-PCIe-40-Gigabit-Ethernet-Network-Connections-Under-Linux-?product=82947" rel="noreferrer" target="_blank">https://downloadcenter.intel.<wbr>com/download/24411/Intel-<wbr>Network-Adapter-Driver-for-<wbr>PCIe-40-Gigabit-Ethernet-<wbr>Network-Connections-Under-<wbr>Linux-?product=82947</a><br>
>>> <br>
>>> <br>
>>> <br>
>>> (Just remember that you will need to repeat this after any kernel updates)<br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> <br>
>>> From: fatema bannatwala [mailto:<a href="mailto:fatema.bannatwala@gmail.com">fatema.bannatwala@<wbr>gmail.com</a>]<br>
>>> Sent: Wednesday, July 11, 2018 13:55 PM<br>
>>> To: Cloherty, Sean E <<a href="mailto:scloherty@mitre.org">scloherty@mitre.org</a>><br>
>>> Cc: <a href="mailto:oisf-users@lists.openinfosecfoundation.org">oisf-users@lists.<wbr>openinfosecfoundation.org</a><br>
>>> Subject: Re: [Oisf-users] High Suricata capture.kernel_drops<br>
>>> <br>
>>> <br>
>>> <br>
>>> Hi Sean,<br>
>>> <br>
>>> <br>
>>> <br>
>>> Thanks for some quick points and recommendations.<br>
>>> <br>
>>> I will work through those, and see if it helps.<br>
>>> <br>
>>> <br>
>>> <br>
>>> The documentation refers the tuning assuming two NICs p1p1 and p1p3, which<br>
>>> was getting me confused, as I only have single NIC with 20 cores and 40<br>
>>> online threads, so was struggling to set the config options right in the<br>
>>> yaml file for cpu_affinity. I will try the hard coded method instead of all<br>
>>> and see if it helps.<br>
>>> <br>
>>> <br>
>>> <br>
>>> Fatema.<br>
>> <br>
>> <br>
>> ______________________________<wbr>_________________<br>
>> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
>> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
>> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
>> <br>
>> Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
>> Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/<wbr>training/</a><br>
> <br>
> <br>
> <br>
> -- <br>
> Regards,<br>
> Peter Manev<br>
> ______________________________<wbr>_________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@<wbr>openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" rel="noreferrer" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" rel="noreferrer" target="_blank">http://suricata-ids.org/<wbr>support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" rel="noreferrer" target="_blank">https://lists.<wbr>openinfosecfoundation.org/<wbr>mailman/listinfo/oisf-users</a><br>
> <br>
> Conference: <a href="https://suricon.net" rel="noreferrer" target="_blank">https://suricon.net</a><br>
> Trainings: <a href="https://suricata-ids.org/training/" rel="noreferrer" target="_blank">https://suricata-ids.org/<wbr>training/</a><br>
</div></div></blockquote></div><br></div>