[Oisf-users] number of alerts versus performance

Fri Jul 1 14:03:07 UTC 2016

I have HP 10 gig fiber nic. Unfortunately it does not have affinity script. I've tried manually assigning each queue to each CPU and I've tried assigning all CPUs to all queues. It seems to only use just one queue/cpu for each physical NIC. I've looked at another identical sensor with higher traffic throughput and it is using two queues for each NIC with irqbalance turned on. I wonder if kernel decided that one queue is enough for 3mil packets a minute.

My guess would be that the packet loss is due to the traffic type and home_net definition. This sensor is in management zone so it has a lot of Active Directory, SSH, RDP, and other management traffic. I dont know if this can have a negative impact on performance.

For example, I have another sensor which mostly processes HTTP/S and SQL/Oracle traffic with 50mil packets a minute and it only has couple of % of packet loss. Sam physical server with the same suricata config with the exception of HOME_NET

________________________________
From: Peter Manev <petermanev at gmail.com>
Sent: Thursday, June 30, 2016 9:57 PM
To: Yasha Zislin
Cc: oisf-users at lists.openinfosecfoundation.org
Subject: Re: [Oisf-users] number of alerts versus performance

On Thu, 2016-06-30 at 17:14 +0000, Yasha Zislin wrote:
> More info. It seems my threads process different amount of packets. It
> is not evenly distributed. Is there a setting somewhere for that in
> Suricata or in PFRING? It seems that thread with 100% cpu utilization
> changes from one to another over time. At that time I notice from
> stats.log that new busy thread is processing more packets.
>

You mentioned earlier you were messing around with a number of diff
settings - might be related. Did you use the irq affinity script (if you
got an Intel nic)?

>
>
>
> ______________________________________________________________________
> From: Peter Manev <petermanev at gmail.com>
> Sent: Thursday, June 30, 2016 4:27 PM
> To: Yasha Zislin
> Cc: oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] number of alerts versus performance
>
> On Thu, 2016-06-30 at 15:54 +0000, Yasha Zislin wrote:
> > Peter,
> >
> >
> > I found one alert that was causing high alert count. After I've
> > disabled it, count went down but packet loss is still around 20%.
> >
> >
> > my stats.log does not contain anything useful such as flow emergency
> > mode, or ssn memcap drop. The only thing that is off is kernel
> drops,
> > and tcp reassembly gaps.
> > From my understanding kernel drops have nothing to do with Suricata
> > and point to OS problems.
> >
> >
> > I do see one of the CPUs peak at 100% when packet loss increases.
> One
> > thing to note. Two other CPUs are working on capturing traffic with
> > high IRQs. My guess would be flow manager or detection engine.
> >
>
>
> You can see if you get more info from:
> top -H -p `pidof suricata`
> and
> perf top -c cpu_number_here
> example: perf top -c 0
>
> > I dunno.
> >
> >
> > Thanks
> >
> >
> >
> >
> >
> ______________________________________________________________________
> > From: Peter Manev <petermanev at gmail.com>
> > Sent: Thursday, June 30, 2016 3:00 PM
> > To: Yasha Zislin
> > Cc: oisf-users at lists.openinfosecfoundation.org
> > Subject: Re: [Oisf-users] number of alerts versus performance
> >
> > On Thu, 2016-06-30 at 14:41 +0000, Yasha Zislin wrote:
> > > I have been trying to figure out a packet loss on one of my
> sensors
> > > and I am puzzled.
> > >
> > > It is has 16 gigs of RAM, one quad core AMD CPU, and nic sees
> about
> > 3
> > > million packets per minute. Nothing special in my mind. I am using
> > > PFRING 6.5.0 and Suricata 3.1.
> > >
> > > I get about 20% to 40% packet loss.  I have another identical
> server
> > > which sees the same amount of traffic and maybe some of the same
> > > traffic as well.
> > >
> > > I've been messing around with NIC settings, IRQs, PFRING settings,
> > > Suricata settings trying to figure out why such a high packet
> loss.
> > >
> > >
> > > I have just realized one big difference in these two sensors.
> > > Problematic one gets 2k to 4k of alerts per minute which sounds
> > huge.
> > >
> >
> > Any particular sig that is alerting in excess ?
> >
> > > Second one gets like 80 alerts per minute. Both have the same
> > > rulesets.
> > >
> > >
> > > The difference of course is the home_net variable.
> > >
> > >
> > > Can the fact that Suricata processes more rules due to HOME_NET
> > > definition cause high performance strain on the server?
> > >
> >
> > Yes HOME_NET size has effect on performance as well (among other
> > things). For example -
> > HOME_NET: "any"
> > EXTERNAL_NET: "any"
> > will certainly degrade your performance.
> >
> > >
> > > If the packet does not match per HOME_NET, it will be discarded
> > before
> > > being processed in rules. Correct?
> > >
> > > Versus if packet passes HOME_NET check, it would have to go
> through
> > > all of the rules, hence cause higher CPU utilization.
> > >
> > >
> > > Thank you for the clarification.
> > >
> > >
> > > _______________________________________________
> > > Suricata IDS Users mailing list:
> > oisf-users at openinfosecfoundation.org
> > > Site: http://suricata-ids.org | Support:
[https://secure.gravatar.com/blavatar/b35fe77e09a7541f738f500f4db6b857?s=200&ts=1467381526]<http://suricata-ids.org/>

Suricata<http://suricata-ids.org/>
suricata-ids.org
Open Source IDS / IPS / NSM engine

>
>
> Suricata
> suricata-ids.org
> Open Source IDS / IPS / NSM engine
>
>
> > http://suricata-ids.org/support/
> >
> >
> > Suricata
> > suricata-ids.org
> > Open Source IDS / IPS / NSM engine
> >
> >
> > > List:
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > Suricata User Conference November 9-11 in Washington, DC:
> > http://oisfevents.net
> >
> >
>
>

--
Regards,
Peter Manev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20160701/0317a27a/attachment-0001.html>