[Oisf-users] number of alerts versus performance

Thu Jul 7 19:59:57 UTC 2016

On 07/07/16 at 01:07, Yasha Zislin wrote:
> I tried using top with "H". I have 4 threads running under suricata.
> One of them peaks at 100% while others are at 50% or so. On occasion
> different thread gets 100%. Packet loss occurs on the thread which has
> 100% utilization. I am not sure why these are not spread out evenly.

Which thread was at 100%?
Might need some finetuning to get better distribution.

> 
> ________________________________ From: Oisf-users
> <oisf-users-bounces at lists.openinfosecfoundation.org> on behalf of
> Andreas Herz <andi at geekosphere.org> Sent: Wednesday, July 6, 2016 8:12
> PM To: oisf-users at lists.openinfosecfoundation.org Subject: Re:
> [Oisf-users] number of alerts versus performance
> 
> On 06/07/16 at 11:06, Yasha Zislin wrote:
> > Sure. Home nets are nothing special.
> >
> > 123.123.64.0/18,123.123.128.0/18
> 
> Well that shouldn't be an issue unless there is a strange bug :)
> 
> >
> > Is there a way to find out which process in Suricata is eating up
> > the CPU? Maybe that's how I can trace it down to what is causing
> > dramatic packet loss.
> 
> You can use top, htop and perf top.  In top use "H" for example
> 
> >
> > Thank you all.
> >
> >
> > ________________________________ From: Oisf-users
> > <oisf-users-bounces at lists.openinfosecfoundation.org> on behalf of
> > Andreas Herz <andi at geekosphere.org> Sent: Tuesday, July 5, 2016 8:20
> > PM To: oisf-users at lists.openinfosecfoundation.org Subject: Re:
> > [Oisf-users] number of alerts versus performance
> >
> > On 01/07/16 at 14:03, Yasha Zislin wrote:
> > > My guess would be that the packet loss is due to the traffic type
> > > and home_net definition. This sensor is in management zone so it
> > > has a lot of Active Directory, SSH, RDP, and other management
> > > traffic. I dont know if this can have a negative impact on
> > > performance.
> >
> > Can you share your HOME_NET definition (with example IPs)?  So far
> > we can just guess :)
> >
> > >
> > > For example, I have another sensor which mostly processes HTTP/S
> > > and SQL/Oracle traffic with 50mil packets a minute and it only has
> > > couple of % of packet loss. Sam physical server with the same
> > > suricata config with the exception of HOME_NET
> > >
> > >
> > > ________________________________ From: Peter Manev
> > > <petermanev at gmail.com> Sent: Thursday, June 30, 2016 9:57 PM To:
> > > Yasha Zislin Cc: oisf-users at lists.openinfosecfoundation.org
> > > Subject: Re: [Oisf-users] number of alerts versus performance
> > >
> > > On Thu, 2016-06-30 at 17:14 +0000, Yasha Zislin wrote:
> > > > More info. It seems my threads process different amount of
> > > > packets.  It is not evenly distributed. Is there a setting
> > > > somewhere for that in Suricata or in PFRING? It seems that
> > > > thread with 100% cpu utilization changes from one to another
> > > > over time. At that time I notice from stats.log that new busy
> > > > thread is processing more packets.
> > > >
> > >
> > > You mentioned earlier you were messing around with a number of
> > > diff settings - might be related. Did you use the irq affinity
> > > script (if you got an Intel nic)?
> > >
> > > >
> > > >
> > > >
> > > > ______________________________________________________________________
> > > > From: Peter Manev <petermanev at gmail.com> Sent: Thursday, June
> > > > 30, 2016 4:27 PM To: Yasha Zislin Cc:
> > > > oisf-users at lists.openinfosecfoundation.org Subject: Re:
> > > > [Oisf-users] number of alerts versus performance
> > > >
> > > > On Thu, 2016-06-30 at 15:54 +0000, Yasha Zislin wrote:
> > > > > Peter,
> > > > >
> > > > >
> > > > > I found one alert that was causing high alert count. After
> > > > > I've disabled it, count went down but packet loss is still
> > > > > around 20%.
> > > > >
> > > > >
> > > > > my stats.log does not contain anything useful such as flow
> > > > > emergency mode, or ssn memcap drop. The only thing that is off
> > > > > is kernel
> > > > drops,
> > > > > and tcp reassembly gaps.  From my understanding kernel drops
> > > > > have nothing to do with Suricata and point to OS problems.
> > > > >
> > > > >
> > > > > I do see one of the CPUs peak at 100% when packet loss
> > > > > increases.
> > > > One
> > > > > thing to note. Two other CPUs are working on capturing traffic
> > > > > with high IRQs. My guess would be flow manager or detection
> > > > > engine.
> > > > >
> > > >
> > > >
> > > > You can see if you get more info from: top -H -p `pidof
> > > > suricata` and perf top -c cpu_number_here example: perf top -c 0
> > > >
> > > > > I dunno.
> > > > >
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > ______________________________________________________________________
> > > > > From: Peter Manev <petermanev at gmail.com> Sent: Thursday, June
> > > > > 30, 2016 3:00 PM To: Yasha Zislin Cc:
> > > > > oisf-users at lists.openinfosecfoundation.org Subject: Re:
> > > > > [Oisf-users] number of alerts versus performance
> > > > >
> > > > > On Thu, 2016-06-30 at 14:41 +0000, Yasha Zislin wrote:
> > > > > > I have been trying to figure out a packet loss on one of my
> > > > sensors
> > > > > > and I am puzzled.
> > > > > >
> > > > > > It is has 16 gigs of RAM, one quad core AMD CPU, and nic
> > > > > > sees
> > > > about
> > > > > 3
> > > > > > million packets per minute. Nothing special in my mind. I am
> > > > > > using PFRING 6.5.0 and Suricata 3.1.
> > > > > >
> > > > > > I get about 20% to 40% packet loss.  I have another
> > > > > > identical
> > > > server
> > > > > > which sees the same amount of traffic and maybe some of the
> > > > > > same traffic as well.
> > > > > >
> > > > > > I've been messing around with NIC settings, IRQs, PFRING
> > > > > > settings, Suricata settings trying to figure out why such a
> > > > > > high packet
> > > > loss.
> > > > > >
> > > > > >
> > > > > > I have just realized one big difference in these two
> > > > > > sensors.  Problematic one gets 2k to 4k of alerts per minute
> > > > > > which sounds
> > > > > huge.
> > > > > >
> > > > >
> > > > > Any particular sig that is alerting in excess ?
> > > > >
> > > > > > Second one gets like 80 alerts per minute. Both have the
> > > > > > same rulesets.
> > > > > >
> > > > > >
> > > > > > The difference of course is the home_net variable.
> > > > > >
> > > > > >
> > > > > > Can the fact that Suricata processes more rules due to
> > > > > > HOME_NET definition cause high performance strain on the
> > > > > > server?
> > > > > >
> > > > >
> > > > > Yes HOME_NET size has effect on performance as well (among
> > > > > other things). For example - HOME_NET: "any" EXTERNAL_NET:
> > > > > "any" will certainly degrade your performance.
> > > > >
> > > > > >
> > > > > > If the packet does not match per HOME_NET, it will be
> > > > > > discarded
> > > > > before
> > > > > > being processed in rules. Correct?
> > > > > >
> > > > > > Versus if packet passes HOME_NET check, it would have to go
> > > > through
> > > > > > all of the rules, hence cause higher CPU utilization.
> > > > > >
> > > > > >
> > > > > > Thank you for the clarification.
> > > > > >
> > > > > >
> > > > > > _______________________________________________ Suricata IDS
> > > > > > Users mailing list:
> > > > > oisf-users at openinfosecfoundation.org
> > > > > > Site: http://suricata-ids.org | Support:
> [https://secure.gravatar.com/blavatar/b35fe77e09a7541f738f500f4db6b857?s=200&ts=1467853530]<http://suricata-ids.org/>
> 
> Suricata<http://suricata-ids.org/> suricata-ids.org Open Source IDS /
> IPS / NSM engine
> 
> 
> 
> > [https://secure.gravatar.com/blavatar/b35fe77e09a7541f738f500f4db6b857?s=200&ts=1467803086]<http://suricata-ids.org/>
> >
> > Suricata<http://suricata-ids.org/> suricata-ids.org Open Source IDS
> > / IPS / NSM engine
> >
> >
> >
> > > [https://secure.gravatar.com/blavatar/b35fe77e09a7541f738f500f4db6b857?s=200&ts=1467381526]<http://suricata-ids.org/>
> > >
> > > Suricata<http://suricata-ids.org/> suricata-ids.org Open Source
> > > IDS / IPS / NSM engine
> > >
> > >
> > >
> > > >
> > > >
> > > > Suricata suricata-ids.org Open Source IDS / IPS / NSM engine
> > > >
> > > >
> > > > > http://suricata-ids.org/support/
> > > > >
> > > > >
> > > > > Suricata suricata-ids.org Open Source IDS / IPS / NSM engine
> > > > >
> > > > >
> > > > > > List:
> > > > > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > > > > Suricata User Conference November 9-11 in Washington, DC:
> > > > > http://oisfevents.net
> > > > >
> > > > >
> > > >
> > > >
> > >
> > > -- Regards, Peter Manev
> > >
> >
> > > _______________________________________________ Suricata IDS Users
> > > mailing list: oisf-users at openinfosecfoundation.org Site:
> > > http://suricata-ids.org | Support:
> > > http://suricata-ids.org/support/ List:
> > > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > > Suricata User Conference November 9-11 in Washington, DC:
> > > http://oisfevents.net
> >
> >
> > -- Andreas Herz _______________________________________________
> > Suricata IDS Users mailing list:
> > oisf-users at openinfosecfoundation.org Site: http://suricata-ids.org |
> > Support: http://suricata-ids.org/support/ List:
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > Suricata User Conference November 9-11 in Washington, DC:
> > http://oisfevents.net
> 
> -- Andreas Herz _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/ List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> Suricata User Conference November 9-11 in Washington, DC:
> http://oisfevents.net

-- 
Andreas Herz