<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yui_3_16_0_1_1444311546408_95844" dir="ltr"><span id="yui_3_16_0_1_1444311546408_96131">Hi Jesper, I just checked and I see stream 0 having the most amount of drops, but the others all have drops with values relatively close together - between 2.3M and 5M each. So it's sounding like your first point may be right. How would you approach the Suricata configuration to address this?</span></div> <br><div class="qtdSeparateBR"><br><br></div><div class="yahoo_quoted" style="display: block;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div dir="ltr"> <font size="2" face="Arial"> On Wednesday, October 14, 2015 7:42 AM, Jesper Lyager Nielsen <jln@napatech.com> wrote:<br> </font> </div> <br><br> <div class="y_msg_container"><div id="yiv9330149765"><div>
Hi Steve.
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">The profiling tool is also good. Do you so equal packet drops on all hostbuffers, or is it just a few hostbuffers that show drops?</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">The reason why I’m asking this is:</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765"> - if all hostbuffers drops approx. equal amount of packets I think it’s a mis-configuration of Suricata. Suricata is not taking of packets fast enough, and probably not threaded correctly to the hostbuffers.</div>
<div class="yiv9330149765"> - if one or two hostbuffers are dropping packets, and some are not, it is probably the Suricata packet analyzer that is too busy. Meaning that too much traffic is coming in on the thread.</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">As previously mentioned it can easily be the case that you have to many filters for the Suricata thread to process.</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">You also have the option to prefilter the hostbuffers in order to ease the traffic on a hostbuffer.</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">Best Regards</div>
<div class="yiv9330149765">Jesper</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765yqt6504996094" id="yiv9330149765yqt77627"><div class="yiv9330149765">
<div>
<blockquote class="yiv9330149765" type="cite">
<div class="yiv9330149765">On 14 Oct 2015, at 12:50, Steve <<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:castle1126@yahoo.com" target="_blank" href="mailto:castle1126@yahoo.com">castle1126@yahoo.com</a>> wrote:</div>
<br clear="none" class="yiv9330149765Apple-interchange-newline">
<div class="yiv9330149765">
<div class="yiv9330149765">
<div class="yiv9330149765">Hi Jesper,</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">Thank you for this. I worked with a local Napatech engineer to use NTPL to set up more streams (10), set up the 5-Tuple index and make sure those streams are tied to Numa 1 because of the interrupts seen in /proc/interrupts. He told me of the
'profiling' command which would show if there are any drops per stream - which there are. I've also been running the monitoring tool, which may be showing drops (I previously posted I had no drops looking at that tool, but I think I was viewing the stats
from that tool incorrectly).</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">So based on your comments, seeing drops in each stream shows Suricata isn't configured appropriately to keep up with the host buffers. I've tried different worker configurations in suricata.yaml but no luck. Same goes for running with "autofp".
Each time I restart Suricata with a reconfigured yaml I see drops via the profiling tool in a short amount of time, and those drops grow in number.</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">Do you have any suggestions on a way to configure?</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">Thanks!!</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
On Oct 14, 2015, at 6:14 AM, Jesper Lyager Nielsen <<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:jln@napatech.com" target="_blank" href="mailto:jln@napatech.com">jln@napatech.com</a>> wrote:<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
</div>
<blockquote class="yiv9330149765" type="cite">
<div class="yiv9330149765">Hi Steve.
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">I think what Victor means is that with a Napatech card you are able to distribute your traffic to different hostbuffers, fx based on a 5 tuple index. You can then have a Suricata thread for each hostbuffer, and hereby distribute your load.</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">First thing I would check in your setup is if your ‘monitoring’ tool (default location: '/opt/napatech3/bin/monitoring’) are showing drops or errors. If that is the case Suricata is not taking packets of the Napatech hostbuffers fast enough.</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765">Best Regards</div>
<div class="yiv9330149765">Jesper</div>
<div class="yiv9330149765"><br clear="none" class="yiv9330149765">
<div class="yiv9330149765">
<blockquote class="yiv9330149765" type="cite">
<div class="yiv9330149765">On 13 Oct 2015, at 19:59, Stephen Castellarin <<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:castle1126@yahoo.com" target="_blank" href="mailto:castle1126@yahoo.com">castle1126@yahoo.com</a>> wrote:</div>
<br clear="none" class="yiv9330149765Apple-interchange-newline">
<div class="yiv9330149765">
<div class="yiv9330149765">
<div class="yiv9330149765" style="background-color:rgb(255, 255, 255);font-family:HelveticaNeue, 'Helvetica Neue', Helvetica, Arial, 'Lucida Grande', sans-serif;font-size:16px;">
<div class="yiv9330149765" id="yiv9330149765yui_3_16_0_1_1444311546408_83104"><span class="yiv9330149765">Hi Victor,</span></div>
<div class="yiv9330149765" id="yiv9330149765yui_3_16_0_1_1444311546408_83104"><span class="yiv9330149765"><br clear="none" class="yiv9330149765">
</span></div>
<div class="yiv9330149765" id="yiv9330149765yui_3_16_0_1_1444311546408_83104"><span class="yiv9330149765" id="yiv9330149765yui_3_16_0_1_1444311546408_83743">I'm trying to understand what you meant by making "sure all packets from a flow are delivered to the same Suricata thread".</span></div>
<div class="yiv9330149765" id="yiv9330149765yui_3_16_0_1_1444311546408_83104"><span class="yiv9330149765"><br clear="none" class="yiv9330149765">
</span></div>
<div class="yiv9330149765" dir="ltr" id="yiv9330149765yui_3_16_0_1_1444311546408_83104"><span class="yiv9330149765" id="yiv9330149765yui_3_16_0_1_1444311546408_83173">Right now I'm looking at the /proc/interrupts and it shows that CPU 1 is handling the interrupts for the Napatech card (based on lscpu NUMA node
1 is handling CPUs 1,3,5,7,9,11,13,15,17,19). I've set the Napatech card to assign its HostBuffersRx to NUMA node 1. Would it be wise to set the CPU affinity for receive, decode and stream-cpu-set to all the CPUs on NUMA node 1? And if so, then should I
assign the detect-cpu-set to the other CPUs on NUMA node 0?<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
Steve</span></div>
<br clear="none" class="yiv9330149765">
<div class="yiv9330149765qtdSeparateBR"><br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
</div>
<div class="yiv9330149765yahoo_quoted" style="display: block;">
<div class="yiv9330149765" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;">
<div class="yiv9330149765" style="font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px;">
<div class="yiv9330149765" dir="ltr"><font class="yiv9330149765" size="2" face="Arial">On Friday, October 9, 2015 12:01 PM, Victor Julien <<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:lists@inliniac.net" target="_blank" href="mailto:lists@inliniac.net">lists@inliniac.net</a>> wrote:<br clear="none" class="yiv9330149765">
</font></div>
<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
<div class="yiv9330149765y_msg_container">On 09-10-15 17:15, Stephen Castellarin wrote:<br clear="none" class="yiv9330149765">
> Yes there still is progress to make. Looking at CPU utilization through<br clear="none" class="yiv9330149765">
> SAR, for today I'm seeing an average of 88.86 %idle, so they're not<br clear="none" class="yiv9330149765">
> being overworked. As far as memory consumption, stats are showing I'm<br clear="none" class="yiv9330149765">
> using roughly 50gb of 128gb available. So I know I have plenty of<br clear="none" class="yiv9330149765">
> breathing room from the hardware's perspective.<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
One thing to check is how the card does the traffic distributions. You<br clear="none" class="yiv9330149765">
need to make sure all packets from a flow are delivered to the same<br clear="none" class="yiv9330149765">
Suricata thread. IIRC napatech cards give you a lot of control there.<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
Cheers,<br clear="none" class="yiv9330149765">
Victor<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
> To your point about the rules, I know I've stripped down a whole bunch<br clear="none" class="yiv9330149765">
> of the ETPRO rules - only sticking with the exploit, malware, scan,<br clear="none" class="yiv9330149765">
> trojan, current_events, web_server and web_specific_apps rules. The<br clear="none" class="yiv9330149765">
> largest number of rules from that list are in the trojan.rules (~9763),<br clear="none" class="yiv9330149765">
> web_specific_apps.rules (~5603) and current_events.rules(~2505). When I<br clear="none" class="yiv9330149765">
> cut down to that list of rule files from the full ETPRO rule list that<br clear="none" class="yiv9330149765">
> definitely cut out unnecessary stuff for us. It's going to be real<br clear="none" class="yiv9330149765">
> tough to dig through the remainder to see what is pertinent to us and<br clear="none" class="yiv9330149765">
> what isn't.<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> On Friday, October 9, 2015 10:32 AM, Rob MacGregor<br clear="none" class="yiv9330149765">
> <<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:rob.macgregor@gmail.com" target="_blank" href="mailto:rob.macgregor@gmail.com">rob.macgregor@gmail.com</a>> wrote:<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> On Fri, Oct 9, 2015 at 3:05 PM Stephen Castellarin <<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:castle1126@yahoo.com" target="_blank" href="mailto:castle1126@yahoo.com">castle1126@yahoo.com</a><br clear="none" class="yiv9330149765">
> <mailto:<a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:castle1126@yahoo.com" target="_blank" href="mailto:castle1126@yahoo.com">castle1126@yahoo.com</a>>> wrote:
<div class="yiv9330149765yqt2922283843" id="yiv9330149765yqtfd51804"><br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> Sorry for the quick reply yeaterday, I forgot to hit Reply All.<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> As for the tuning, I know my current, underpowered Suricata system<br clear="none" class="yiv9330149765">
> is missing events, as is my new hardware/configuration. I base this<br clear="none" class="yiv9330149765">
> on some attack traffic I saw from one IP yesterday. <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> So our configuration is a front end router feeding an inline IPS<br clear="none" class="yiv9330149765">
> which then is tapped - one tap to my old Suricata system and the<br clear="none" class="yiv9330149765">
> second to my new Suricata system. From a full take packet capture I<br clear="none" class="yiv9330149765">
> see 45 attempts to issue malicious POST attempts to a webserver we<br clear="none" class="yiv9330149765">
> have. My original Suricata system triggered on 10 of those while my<br clear="none" class="yiv9330149765">
> new Suricata triggered on 15. I then took the pcap I extracted and<br clear="none" class="yiv9330149765">
> ran it through Suricata on the new system and that system showed it<br clear="none" class="yiv9330149765">
> trigger on all 45. So that's giving me a feeling that I'm not<br clear="none" class="yiv9330149765">
> tuning something correct - causing the running Suricata to miss things.<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> So, things are improving but there's still progress to make?<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> I'd look at things like CPU and RAM usage - are you maxing out your<br clear="none" class="yiv9330149765">
> CPUs/RAM?<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> Also, really look at those rules, are they really all relevant to your<br clear="none" class="yiv9330149765">
> network? Also, if you strip it down to just the rules that'd catch those<br clear="none" class="yiv9330149765">
> POST attempts, does it fire for every event?<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> -- <br clear="none" class="yiv9330149765">
> Rob </div>
<br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
> _______________________________________________<br clear="none" class="yiv9330149765">
> Suricata IDS Users mailing list: <a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:oisf-users@openinfosecfoundation.org" target="_blank" href="mailto:oisf-users@openinfosecfoundation.org">
oisf-users@openinfosecfoundation.org</a><br clear="none" class="yiv9330149765">
> Site: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://suricata-ids.org/">
http://suricata-ids.org </a>| Support: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://suricata-ids.org/support/">
http://suricata-ids.org/support/</a><br clear="none" class="yiv9330149765">
> List: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users">
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br clear="none" class="yiv9330149765">
> Suricata User Conference November 4 & 5 in Barcelona: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://oisfevents.net/">
http://oisfevents.net</a><br clear="none" class="yiv9330149765">
> <br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
-- <br clear="none" class="yiv9330149765">
---------------------------------------------<br clear="none" class="yiv9330149765">
Victor Julien<br clear="none" class="yiv9330149765">
<a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://www.inliniac.net/">http://www.inliniac.net/</a><br clear="none" class="yiv9330149765">
PGP: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://www.inliniac.net/victorjulien.asc">
http://www.inliniac.net/victorjulien.asc</a><br clear="none" class="yiv9330149765">
---------------------------------------------<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
_______________________________________________<br clear="none" class="yiv9330149765">
Suricata IDS Users mailing list: <a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:oisf-users@openinfosecfoundation.org" target="_blank" href="mailto:oisf-users@openinfosecfoundation.org">
oisf-users@openinfosecfoundation.org</a><br clear="none" class="yiv9330149765">
Site: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://suricata-ids.org/">http://suricata-ids.org
</a>| Support: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://suricata-ids.org/support/">
http://suricata-ids.org/support/</a><br clear="none" class="yiv9330149765">
List: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users">
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br clear="none" class="yiv9330149765">
Suricata User Conference November 4 & 5 in Barcelona: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://oisfevents.net/">
http://oisfevents.net</a>
<div class="yiv9330149765yqt2922283843" id="yiv9330149765yqtfd46506"><br clear="none" class="yiv9330149765">
</div>
<br clear="none" class="yiv9330149765">
<br clear="none" class="yiv9330149765">
</div>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br clear="none" class="yiv9330149765">
Suricata IDS Users mailing list: <a rel="nofollow" shape="rect" class="yiv9330149765" ymailto="mailto:oisf-users@openinfosecfoundation.org" target="_blank" href="mailto:oisf-users@openinfosecfoundation.org">
oisf-users@openinfosecfoundation.org</a><br clear="none" class="yiv9330149765">
Site: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://suricata-ids.org/">http://suricata-ids.org</a> | Support:
<a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://suricata-ids.org/support/">http://suricata-ids.org/support/</a><br clear="none" class="yiv9330149765">
List: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users">
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br clear="none" class="yiv9330149765">
Suricata User Conference November 4 & 5 in Barcelona: <a rel="nofollow" shape="rect" class="yiv9330149765" target="_blank" href="http://oisfevents.net/">
http://oisfevents.net</a></div>
</blockquote>
</div>
<br clear="none" class="yiv9330149765">
</div>
Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please
notify the sender immediately and delete this e-mail from your system. </div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br clear="none" class="yiv9330149765">
</div></div>
Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please
notify the sender immediately and delete this e-mail from your system.
</div></div><br><br></div> </div> </div> </div></div></body></html>