<div dir="ltr">Peter, <div><br></div><div>I reviewed my compile of Suricata 4.0.3 I noticed that I was using Hyperscan version 4.7, as opposed to version 4.2 noted in the Suricata documentation (<a href="http://suricata.readthedocs.io/en/latest/performance/hyperscan.html">http://suricata.readthedocs.io/en/latest/performance/hyperscan.html</a>). After recompiling with 4.2 I was able to get Suricata 4.0.3 to run for 42 minutes before it started dropping packets uncontrollably. </div><div><br></div><div>I then made a change to /proc/sys/vm/max_map_count based on a note in Napatech's documentation: "Especially for large host buffer configurations it is necessary to
adjust the kernel sysctl "vm.max_map_count" (/proc/sys/vm/max_map_count). The kernel sysctl "vm.max_map_count" (/proc/sys/vm/max_map_count)
should be adjusted to (at least) the total configured host buffer memory
in MB multiplied by four.<br></div><div>
Example for total host buffer size 128GB (131072MB): 131072*4 = 524288.
Hence the minimum value for "vm.max_map_count" is 524288."<div><br></div><div>In my case I'm using 17 host buffers at 2048MB per ((17 * 2048) * 4), which would be 139264. My vm.max_map_count previously was 65530 (I guess default for Ubuntu 14.04). After changing that and re-running Suricata 4.0.3 it ran for 45 minutes before the buffer/CPU issue came back.</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 23, 2018 at 9:49 AM, Steve Castellarin <span dir="ltr"><<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr" style="font-size:12.8px">Hi Peter,<div><br></div><div>I just realized I responded directly to you instead of the mailing list - so here's my response, updated.</div><div><br></div><div>I made a change to my YAML file for 4.0.3, dropping the detect-thread-ratio from 1.5 to 1 and on Friday was able to run Suricata 4.0.3 for five hours before the issue occurred. This run did handle sustained network traffic of 1.2 through 1.7gbps. So that is a step in the positive direction. I'm going to have a hard time running 4.0.3 without rules, as this unfortunately is our only Suricata instance running our rule set.</div><div><br></div><div>I've noticed one thing that's strange. In my YAML file I have the "autofp-scheduler" set to "active-packets". Yet everytime I run Suricata I see this noted in suricata.log "using flow hash instead of active packets". When I comment out the "autofp-scheduler" setting in the YAML file then that message disappears. Any idea on what that is all about?</div></div><div class="m_-6637883427268883856gmail-yj6qo m_-6637883427268883856gmail-ajU" style="font-size:12.8px"></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jan 21, 2018 at 3:49 PM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><span><br><div><br>On 18 Jan 2018, at 19:21, Steve Castellarin <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br><br></div><blockquote type="cite"><div><div dir="ltr">And also, the bandwidth utilization was just over 800Mbps.</div><div class="gmail_extra"><br></div></div></blockquote><div><br></div></span><div>Can you try the same run but this time - load no rules. I would like to see if it would make difference or not in the same amount of time.</div><div><div class="m_-6637883427268883856h5"><div><br></div><br><blockquote type="cite"><div><div class="gmail_extra"><div class="gmail_quote">On Thu, Jan 18, 2018 at 1:16 PM, Steve Castellarin <span dir="ltr"><<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Peter,<div><br>Those changes didn't help. Around 23+ minutes into the run one worker CPU (#30) stayed at 100% while buffer NT11 dropped packets and would not recover. I'm attaching a zip file that has the stats.log for that run, the suricata.log file as well as the information seen at the command line after issuing "/usr/bin/suricata -vvv -c /etc/suricata/suricata.yaml --napatech --runmode workers -D".<span class="m_-6637883427268883856m_-4746930007647200886HOEnZb"><font color="#888888"><br><br>Steve</font></span></div><div><br></div></div><div class="m_-6637883427268883856m_-4746930007647200886HOEnZb"><div class="m_-6637883427268883856m_-4746930007647200886h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 18, 2018 at 11:30 AM, Steve Castellarin <span dir="ltr"><<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">We never see above 2Gbps. When the issue occurred a little bit ago I was running the Napatech "monitoring" tool and it was saying we were between 650-900Mbps. I'll note the bandwidth utilization when the next issue occurs.</div><div class="m_-6637883427268883856m_-4746930007647200886m_-2355543287138870934HOEnZb"><div class="m_-6637883427268883856m_-4746930007647200886m_-2355543287138870934h5"><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jan 18, 2018 at 11:28 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>On Thu, Jan 18, 2018 at 5:27 PM, Steve Castellarin<br>
<<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
> When you mean the "size of the traffic", are you asking what the bandwidth<br>
> utilization is at the time the issue begins?<br>
<br>
</span>Sorry - i mean the traffic you sniff - 1/5/10...Gbps ?<br>
<div class="m_-6637883427268883856m_-4746930007647200886m_-2355543287138870934m_1503848611840549081HOEnZb"><div class="m_-6637883427268883856m_-4746930007647200886m_-2355543287138870934m_1503848611840549081h5"><br>
><br>
> I will set things up and send you any/all output after the issue starts.<br>
><br>
> On Thu, Jan 18, 2018 at 11:17 AM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>> wrote:<br>
>><br>
>> On Thu, Jan 18, 2018 at 4:43 PM, Steve Castellarin<br>
>> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> > Hey Peter,<br>
>> ><br>
>> > I tried as you asked. Less than 15 minutes after I restarted Suricata I<br>
>> > saw<br>
>> > my first CPU hitting 100% and one host buffer dropping all packets.<br>
>> > Shortly<br>
>> > after that the second CPU hit 100% and a second host buffer began<br>
>> > dropping<br>
>> > all packets. I'm attaching the stats.log where you'll see at 10:31:11<br>
>> > the<br>
>> > first host buffer (nt1.drop) starts to register dropped packets, then at<br>
>> > 10:31:51 you'll see host buffer nt6.drop begin to register dropped<br>
>> > packets.<br>
>> > At that point I issued the kill.<br>
>> ><br>
>><br>
>> What is the size of the traffic?<br>
>> Can you also try<br>
>> detect:<br>
>> - profile: high<br>
>><br>
>> (as opposed to "custom")<br>
>><br>
>> Also if can run it in verbose mode (-vvv) and send me that compete<br>
>> output after you start having the issues.<br>
>><br>
>> Thanks<br>
>><br>
>><br>
>><br>
>> > Steve<br>
>> ><br>
>> > On Thu, Jan 18, 2018 at 10:05 AM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>><br>
>> > wrote:<br>
>> >><br>
>> >> On Wed, Jan 17, 2018 at 1:29 PM, Steve Castellarin<br>
>> >> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> >> > Hey Pete,<br>
>> >> ><br>
>> >> > Here's the YAML file from the last time I attempted to run 4.0.3 -<br>
>> >> > with<br>
>> >> > the<br>
>> >> > network information removed. Let me know if you need anything else<br>
>> >> > from<br>
>> >> > our<br>
>> >> > configuration. I'll also go to the redmine site to open a bug<br>
>> >> > report.<br>
>> >> ><br>
>> >> > Steve<br>
>> >><br>
>> >> Hi Steve,<br>
>> >><br>
>> >> Can you try without -<br>
>> >><br>
>> >> midstream: true<br>
>> >> asyn-oneside:true<br>
>> >> so<br>
>> >> #midstream: true<br>
>> >> #asyn-oneside:true<br>
>> >><br>
>> >> and lower the "prealloc-session: 1000000" to 100 000 for example<br>
>> >><br>
>> >><br>
>> >> Thank you.<br>
>> >><br>
>> >> ><br>
>> >> > On Wed, Jan 17, 2018 at 6:36 AM, Peter Manev <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>><br>
>> >> > wrote:<br>
>> >> >><br>
>> >> >> On Tue, Jan 16, 2018 at 4:12 PM, Steve Castellarin<br>
>> >> >> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> >> >> > Hey Peter, I didn't know if you had a chance to look at the stats<br>
>> >> >> > log<br>
>> >> >> > and<br>
>> >> >> > configuration file I sent. So far, running 3.1.1 with the updated<br>
>> >> >> > Napatech<br>
>> >> >> > drivers my system is running without any issues.<br>
>> >> >> ><br>
>> >> >><br>
>> >> >> The toughest part of the troubleshooting is that i dont have the set<br>
>> >> >> up to reproduce this.<br>
>> >> >> I didn't see anything that could lead me to definitive conclusion<br>
>> >> >> from<br>
>> >> >> the stats log.<br>
>> >> >> Can you please open a bug report on our redmine with the details<br>
>> >> >> form<br>
>> >> >> this mialthread?<br>
>> >> >><br>
>> >> >> Would it be possible to share the suricata.yaml (privately if you<br>
>> >> >> would like works too; remove all networks)?<br>
>> >> >><br>
>> >> >> Thank you<br>
>> >> >><br>
>> >> >> > On Thu, Jan 11, 2018 at 12:54 PM, Steve Castellarin<br>
>> >> >> > <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> >> >> >><br>
>> >> >> >> Here is the zipped stats.log. I restarted the Napatech drivers<br>
>> >> >> >> before<br>
>> >> >> >> running Suricata 4.0.3 to clear out any previous drop counters,<br>
>> >> >> >> etc.<br>
>> >> >> >><br>
>> >> >> >> The first time I saw a packet drop was at the 12:20:51 mark, and<br>
>> >> >> >> you'll<br>
>> >> >> >> see "nt12.drop" increment. During this time one of the CPUs<br>
>> >> >> >> acting<br>
>> >> >> >> as<br>
>> >> >> >> a<br>
>> >> >> >> "worker" was at 100%. But these drops recovered at the 12:20:58<br>
>> >> >> >> mark,<br>
>> >> >> >> where<br>
>> >> >> >> "nt12.drop" stays constant at 13803. The big issue triggered at<br>
>> >> >> >> the<br>
>> >> >> >> 12:27:05 mark in the file - where one worker CPU was stuck at<br>
>> >> >> >> 100%<br>
>> >> >> >> followed<br>
>> >> >> >> by packet drops in host buffer "nt3.drop". Then came a second<br>
>> >> >> >> CPU<br>
>> >> >> >> at<br>
>> >> >> >> 100%<br>
>> >> >> >> (another "worker" CPU) and packet drops in buffer "nt2.drop" at<br>
>> >> >> >> 12:27:33. I<br>
>> >> >> >> finally killed Suricata just before 12:27:54, where you see all<br>
>> >> >> >> host<br>
>> >> >> >> buffers<br>
>> >> >> >> beginning to drop packets.<br>
>> >> >> >><br>
>> >> >> >> I'm also including the output from the "suricata --dump-config"<br>
>> >> >> >> command.<br>
>> >> >> >><br>
>> >> >> >> On Thu, Jan 11, 2018 at 11:40 AM, Peter Manev<br>
>> >> >> >> <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>><br>
>> >> >> >> wrote:<br>
>> >> >> >>><br>
>> >> >> >>> On Thu, Jan 11, 2018 at 8:02 AM, Steve Castellarin<br>
>> >> >> >>> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> >> >> >>> > Peter, yes that is correct. I worked for almost a couple<br>
>> >> >> >>> > weeks<br>
>> >> >> >>> > with<br>
>> >> >> >>> > Napatech support and they believed the Napatech setup<br>
>> >> >> >>> > (ntservice.ini<br>
>> >> >> >>> > and<br>
>> >> >> >>> > custom NTPL script) are working as they should.<br>
>> >> >> >>> ><br>
>> >> >> >>><br>
>> >> >> >>> Ok.<br>
>> >> >> >>><br>
>> >> >> >>> One major difference between Suricata 3.x and 4.0.x in terms of<br>
>> >> >> >>> Napatech is that they did update the code, some fixes and<br>
>> >> >> >>> updated<br>
>> >> >> >>> the<br>
>> >> >> >>> counters.<br>
>> >> >> >>> There were a bunch of upgrades in Suricata too.<br>
>> >> >> >>> Is it possible to send over a stats.log - when the issue starts<br>
>> >> >> >>> occuring?<br>
>> >> >> >>><br>
>> >> >> >>><br>
>> >> >> >>> > On Thu, Jan 11, 2018 at 9:52 AM, Peter Manev<br>
>> >> >> >>> > <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>><br>
>> >> >> >>> > wrote:<br>
>> >> >> >>> >><br>
>> >> >> >>> >> I<br>
>> >> >> >>> >><br>
>> >> >> >>> >> On 11 Jan 2018, at 07:19, Steve Castellarin<br>
>> >> >> >>> >> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>><br>
>> >> >> >>> >> wrote:<br>
>> >> >> >>> >><br>
>> >> >> >>> >> After my last email yesterday I decided to go back to our<br>
>> >> >> >>> >> 3.1.1<br>
>> >> >> >>> >> install of<br>
>> >> >> >>> >> Suricata, with<br>
>> >> >> >>> >><br>
>> >> >> >>> >><br>
>> >> >> >>> >> the upgraded Napatech version. Since then I've seen no<br>
>> >> >> >>> >> packets<br>
>> >> >> >>> >> dropped<br>
>> >> >> >>> >> with sustained bandwidth of between 1 and 1.7Gbps. So I'm<br>
>> >> >> >>> >> not<br>
>> >> >> >>> >> sure<br>
>> >> >> >>> >> what is<br>
>> >> >> >>> >> going on with my configuration/setup of Suricata 4.0.3.<br>
>> >> >> >>> >><br>
>> >> >> >>> >><br>
>> >> >> >>> >><br>
>> >> >> >>> >> So the only thing that you changed is the upgrade of the<br>
>> >> >> >>> >> Napatech<br>
>> >> >> >>> >> drivers<br>
>> >> >> >>> >> ?<br>
>> >> >> >>> >> The Suricata config stayed the same - you just upgraded to<br>
>> >> >> >>> >> 4.0.3<br>
>> >> >> >>> >> (from<br>
>> >> >> >>> >> 3.1.1) and the observed effect was - after a while all (or<br>
>> >> >> >>> >> most)<br>
>> >> >> >>> >> cpus<br>
>> >> >> >>> >> get<br>
>> >> >> >>> >> pegged at 100% - is that correct ?<br>
>> >> >> >>> >><br>
>> >> >> >>> >><br>
>> >> >> >>> >> On Wed, Jan 10, 2018 at 4:46 PM, Steve Castellarin<br>
>> >> >> >>> >> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>> Hey Peter, no there is no error messages.<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>> On Jan 10, 2018 4:37 PM, "Peter Manev"<br>
>> >> >> >>> >>> <<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>><br>
>> >> >> >>> >>> wrote:<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>> On Wed, Jan 10, 2018 at 11:29 AM, Steve Castellarin<br>
>> >> >> >>> >>> <<a href="mailto:steve.castellarin@gmail.com" target="_blank">steve.castellarin@gmail.com</a>> wrote:<br>
>> >> >> >>> >>> > Hey Peter,<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>> Are there any errors msgs in suricata.log when that happens<br>
>> >> >> >>> >>> ?<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>> Thank you<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>><br>
>> >> >> >>> >>><br>
>> >> >> >>> >>> --<br>
>> >> >> >>> >>> Regards,<br>
>> >> >> >>> >>> Peter Manev<br>
>> >> >> >>> >>><br>
>> >> >> >>> >>><br>
>> >> >> >>> >><br>
>> >> >> >>> ><br>
>> >> >> >>><br>
>> >> >> >>><br>
>> >> >> >>><br>
>> >> >> >>> --<br>
>> >> >> >>> Regards,<br>
>> >> >> >>> Peter Manev<br>
>> >> >> >><br>
>> >> >> >><br>
>> >> >> ><br>
>> >> >><br>
>> >> >><br>
>> >> >><br>
>> >> >> --<br>
>> >> >> Regards,<br>
>> >> >> Peter Manev<br>
>> >> ><br>
>> >> ><br>
>> >><br>
>> >><br>
>> >><br>
>> >> --<br>
>> >> Regards,<br>
>> >> Peter Manev<br>
>> ><br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Regards,<br>
>> Peter Manev<br>
><br>
><br>
<br>
<br>
<br>
</div></div><span class="m_-6637883427268883856m_-4746930007647200886m_-2355543287138870934m_1503848611840549081HOEnZb"><font color="#888888">--<br>
Regards,<br>
Peter Manev<br>
</font></span></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>
</div></blockquote></div></div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>