[Oisf-users] Help with good configuration for Suricata install with Napatech card

Wed Oct 14 12:03:30 UTC 2015

Hi Jesper, I just checked and I see stream 0 having the most amount of drops, but the others all have drops with values relatively close together - between 2.3M and 5M each.  So it's sounding like your first point may be right.  How would you approach the Suricata configuration to address this? 

     On Wednesday, October 14, 2015 7:42 AM, Jesper Lyager Nielsen <jln at napatech.com> wrote:

 Hi Steve.
The profiling tool is also good. Do you so equal packet drops on all hostbuffers, or is it just a few hostbuffers that show drops?
The reason why I’m asking this is:
 - if all hostbuffers drops approx. equal amount of packets I think it’s a mis-configuration of Suricata. Suricata is not taking of packets fast enough, and probably not threaded correctly to the hostbuffers. - if one or two hostbuffers are dropping packets, and some are not, it is probably the Suricata packet analyzer that is too busy. Meaning that too much traffic is coming in on the thread.
As previously mentioned it can easily be the case that you have to many filters for the Suricata thread to process.
You also have the option to prefilter the hostbuffers in order to ease the traffic on a hostbuffer.
Best RegardsJesper

On 14 Oct 2015, at 12:50, Steve <castle1126 at yahoo.com> wrote:
Hi Jesper,
Thank you for this.  I worked with a local Napatech engineer to use NTPL to  set up more streams (10), set up the 5-Tuple index and make sure those streams are tied to Numa 1 because of the interrupts seen in /proc/interrupts.  He told me of the 'profiling' command which would show if there are any drops per stream - which there are.  I've also been running the monitoring tool, which may be showing drops (I previously posted I had no drops looking at that tool, but I think I was viewing the stats from that tool incorrectly).
So based on your comments, seeing drops in each stream shows Suricata isn't configured appropriately to keep up with the host buffers.  I've tried different worker configurations in suricata.yaml but no luck.  Same goes for running with "autofp".  Each time I restart Suricata with a reconfigured yaml I see drops via the profiling tool in a short amount of time, and those drops grow in number.
Do you have any suggestions on a way to configure?
Thanks!!
On Oct 14, 2015, at 6:14 AM, Jesper Lyager Nielsen <jln at napatech.com> wrote:

Hi Steve.
I think what Victor means is that with a Napatech card you are able to distribute your traffic to different hostbuffers, fx based on a 5 tuple index. You can then have a Suricata thread for each hostbuffer, and hereby distribute your load.
First thing I would check in your setup is if your ‘monitoring’ tool (default location: '/opt/napatech3/bin/monitoring’) are showing drops or errors. If that is the case Suricata is not taking packets of the Napatech hostbuffers fast enough.
Best RegardsJesper

On 13 Oct 2015, at 19:59, Stephen Castellarin <castle1126 at yahoo.com> wrote:
Hi Victor,
I'm trying to understand what you meant by making "sure all packets from a flow are delivered to the same Suricata thread".
Right now I'm looking at the /proc/interrupts and it shows that CPU 1 is handling the interrupts for the Napatech card (based on lscpu NUMA node 1 is handling CPUs 1,3,5,7,9,11,13,15,17,19).  I've set the Napatech card to assign its HostBuffersRx to NUMA node 1.  Would it be wise to set the CPU affinity for receive, decode and stream-cpu-set to all the CPUs on NUMA node 1?  And if so, then should I assign the detect-cpu-set to the other CPUs on NUMA node 0?

Steve

On Friday, October 9, 2015 12:01 PM, Victor Julien <lists at inliniac.net> wrote:

On 09-10-15 17:15, Stephen Castellarin wrote:
> Yes there still is progress to make.  Looking at CPU utilization through
> SAR, for today I'm seeing an average of 88.86 %idle, so they're not
> being overworked.  As far as memory consumption, stats are showing I'm
> using roughly 50gb of 128gb available.  So I know I have plenty of
> breathing room from the hardware's perspective.

One thing to check is how the card does the traffic distributions. You
need to make sure all packets from a flow are delivered to the same
Suricata thread. IIRC napatech cards give you a lot of control there.

Cheers,
Victor

> To your point about the rules, I know I've stripped down a whole bunch
> of the ETPRO rules - only sticking with the exploit, malware, scan,
> trojan, current_events, web_server and web_specific_apps rules.  The
> largest number of rules from that list are in the trojan.rules (~9763),
> web_specific_apps.rules (~5603) and current_events.rules(~2505).  When I
> cut down to that list of rule files from the full ETPRO rule list that
> definitely cut out unnecessary stuff for us.  It's going to be real
> tough to dig through the remainder to see what is pertinent to us and
> what isn't.
> 
> 
> 
> On Friday, October 9, 2015 10:32 AM, Rob MacGregor
> <rob.macgregor at gmail.com> wrote:
> 
> 
> On Fri, Oct 9, 2015 at 3:05 PM Stephen Castellarin <castle1126 at yahoo.com
> <mailto:castle1126 at yahoo.com>> wrote:
> 
>    Sorry for the quick reply yeaterday, I forgot to hit Reply All.
> 
>    As for the tuning, I know my current, underpowered Suricata system
>    is missing events, as is my new hardware/configuration.  I base this
>    on some attack traffic I saw from one IP yesterday.  
> 
>    So our configuration is a front end router feeding an inline IPS
>    which then is tapped - one tap to my old Suricata system and the
>    second to my new Suricata system.  From a full take packet capture I
>    see 45 attempts to issue malicious POST attempts to a webserver we
>    have.  My original Suricata system triggered on 10 of those while my
>    new Suricata triggered on 15.  I then took the pcap I extracted and
>    ran it through Suricata on the new system and that system showed it
>    trigger on all 45.  So that's giving me a feeling that I'm not
>    tuning something correct - causing the running Suricata to miss things.
> 
> 
> So, things are improving but there's still progress to make?
> 
> I'd look at things like CPU and RAM usage - are you maxing out your
> CPUs/RAM?
> 
> Also, really look at those rules, are they really all relevant to your
> network? Also, if you strip it down to just the rules that'd catch those
> POST attempts, does it fire for every event?
> 
> -- 
>  Rob 
> 
> 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net
> 

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------

_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
Site: http://suricata-ids.org| Support: http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net

_______________________________________________
Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
Site: http://suricata-ids.org | Support:http://suricata-ids.org/support/
List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
Suricata User Conference November 4 & 5 in Barcelona: http://oisfevents.net

Disclaimer: This email and any files transmitted with it may contain confidential information intended for the addressee(s) only. The information is not to be surrendered or copied to unauthorized persons. If you have received this communication in error, please notify the sender immediately and delete this e-mail from your system. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20151014/aef4c9a2/attachment-0002.html>