[Oisf-users] number of alerts versus performance

[Peter Manev]

On Thu, 2016-06-30 at 17:14 +0000, Yasha Zislin wrote:
> More info. It seems my threads process different amount of packets. It
> is not evenly distributed. Is there a setting somewhere for that in
> Suricata or in PFRING? It seems that thread with 100% cpu utilization
> changes from one to another over time. At that time I notice from
> stats.log that new busy thread is processing more packets.
> 

You mentioned earlier you were messing around with a number of diff
settings - might be related. Did you use the irq affinity script (if you
got an Intel nic)?

> 
> 
> 
> ______________________________________________________________________
> From: Peter Manev <petermanev at gmail.com>
> Sent: Thursday, June 30, 2016 4:27 PM
> To: Yasha Zislin
> Cc: oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] number of alerts versus performance 
>  
> On Thu, 2016-06-30 at 15:54 +0000, Yasha Zislin wrote:
> > Peter,
> > 
> > 
> > I found one alert that was causing high alert count. After I've
> > disabled it, count went down but packet loss is still around 20%.
> > 
> > 
> > my stats.log does not contain anything useful such as flow emergency
> > mode, or ssn memcap drop. The only thing that is off is kernel
> drops,
> > and tcp reassembly gaps. 
> > From my understanding kernel drops have nothing to do with Suricata
> > and point to OS problems.
> > 
> > 
> > I do see one of the CPUs peak at 100% when packet loss increases.
> One
> > thing to note. Two other CPUs are working on capturing traffic with
> > high IRQs. My guess would be flow manager or detection engine.
> > 
> 
> 
> You can see if you get more info from:
> top -H -p `pidof suricata`
> and
> perf top -c cpu_number_here
> example: perf top -c 0
> 
> > I dunno.
> > 
> > 
> > Thanks
> > 
> > 
> > 
> > 
> >
> ______________________________________________________________________
> > From: Peter Manev <petermanev at gmail.com>
> > Sent: Thursday, June 30, 2016 3:00 PM
> > To: Yasha Zislin
> > Cc: oisf-users at lists.openinfosecfoundation.org
> > Subject: Re: [Oisf-users] number of alerts versus performance 
> >  
> > On Thu, 2016-06-30 at 14:41 +0000, Yasha Zislin wrote:
> > > I have been trying to figure out a packet loss on one of my
> sensors
> > > and I am puzzled.
> > > 
> > > It is has 16 gigs of RAM, one quad core AMD CPU, and nic sees
> about
> > 3
> > > million packets per minute. Nothing special in my mind. I am using
> > > PFRING 6.5.0 and Suricata 3.1.
> > > 
> > > I get about 20% to 40% packet loss.  I have another identical
> server
> > > which sees the same amount of traffic and maybe some of the same
> > > traffic as well.
> > > 
> > > I've been messing around with NIC settings, IRQs, PFRING settings,
> > > Suricata settings trying to figure out why such a high packet
> loss.
> > > 
> > > 
> > > I have just realized one big difference in these two sensors.
> > > Problematic one gets 2k to 4k of alerts per minute which sounds
> > huge.
> > > 
> > 
> > Any particular sig that is alerting in excess ?
> > 
> > > Second one gets like 80 alerts per minute. Both have the same
> > > rulesets.
> > > 
> > > 
> > > The difference of course is the home_net variable.
> > > 
> > > 
> > > Can the fact that Suricata processes more rules due to HOME_NET
> > > definition cause high performance strain on the server? 
> > > 
> > 
> > Yes HOME_NET size has effect on performance as well (among other
> > things). For example - 
> > HOME_NET: "any"
> > EXTERNAL_NET: "any"
> > will certainly degrade your performance.
> > 
> > > 
> > > If the packet does not match per HOME_NET, it will be discarded
> > before
> > > being processed in rules. Correct?
> > > 
> > > Versus if packet passes HOME_NET check, it would have to go
> through
> > > all of the rules, hence cause higher CPU utilization.
> > > 
> > > 
> > > Thank you for the clarification.
> > > 
> > > 
> > > _______________________________________________
> > > Suricata IDS Users mailing list:
> > oisf-users at openinfosecfoundation.org
> > > Site: http://suricata-ids.org | Support: 
> 
> 
> Suricata
> suricata-ids.org
> Open Source IDS / IPS / NSM engine
> 
> 
> > http://suricata-ids.org/support/ 
> > 
> > 
> > Suricata
> > suricata-ids.org
> > Open Source IDS / IPS / NSM engine
> > 
> > 
> > > List:
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > Suricata User Conference November 9-11 in Washington, DC:
> > http://oisfevents.net
> > 
> > 
> 
> 

-- 
Regards,
Peter Manev