[Oisf-users] EXTERNAL: Re: unusual packet loss

[Rasmor, Zachary R]

Are you able to characterize the traffic seen by this sensor? With a static
load balancing scheme like cluster_flow, which I believe uses something like
a hash of the TCP-quad to determine which thread processes a flow, it's
certainly possible for a single thread to get a disproportionate amount of
traffic and become bogged down. I've seen it before, though not quite as
dramatic as in your case, since your other threads appear nearly idle. 

 

While the workers runmode is generally accepted as the best practice, you
might play around with autofp or different cluster modes just to see if you
observe any different behavior. You may not want to make these your
permanent settings, but it may help you narrow down your issue.

 

________________________

Zach Rasmor

Email:  <mailto:zachary.r.rasmor at lmco.com> zachary.r.rasmor at lmco.com

Office: 301.240.6116

 

From: Oisf-users [mailto:oisf-users-bounces at lists.openinfosecfoundation.org]
On Behalf Of Yasha Zislin
Sent: Thursday, December 24, 2015 7:09 AM
To: Peter Manev
Cc: oisf-users at lists.openinfosecfoundation.org
Subject: EXTERNAL: Re: [Oisf-users] unusual packet loss

 

I have 4 threads running to monitor one interface. One of the threads is
consuming 100% CPU and starts to have packet loss. Other 3 have zero packet
loss.

> Date: Wed, 23 Dec 2015 22:36:44 +0100
> Subject: Re: [Oisf-users] unusual packet loss
> From: petermanev at gmail.com <mailto:petermanev at gmail.com> 
> To: coolyasha at hotmail.com <mailto:coolyasha at hotmail.com> 
> CC: oisf-users at lists.openinfosecfoundation.org
<mailto:oisf-users at lists.openinfosecfoundation.org> 
> 
> On Wed, Dec 23, 2015 at 3:36 PM, Yasha Zislin <coolyasha at hotmail.com
<mailto:coolyasha at hotmail.com> > wrote:
> > I am running Suricata 2.1beta4 with PF_RING.
> > I have 4 threads (4 logical CPUs) monitoring one interface. After a few
> > minutes of running, I get 50% packet loss.
> > I have tweaked all of the stream reassembly buffers to avoid packet
loss.
> > Only one of the threads gets kernel packet drops. I've noticed that one
CPU
> > is running at 100% and others are almost idle. Looking at stats.log,
that
> > one thread for some reason is digesting more packets than others.
> > Throughput on this sensor is not that big. About 500k packets a minute.
I
> > use this image on other sensors without issues.
> >
> > Need help to figure out why only one thread is doing MOST of the work.
> 
> Can you share "top -H" screenshot ?
> 
> >
> > Thank you.
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
<mailto:oisf-users at openinfosecfoundation.org> 
> > Site: http://suricata-ids.org | Support:
http://suricata-ids.org/support/
> > List:
https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > Suricata User Conference November 4 & 5 in Barcelona:
http://oisfevents.net
> 
> 
> 
> -- 
> Regards,
> Peter Manev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20151225/8061ad58/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 11767 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20151225/8061ad58/attachment-0002.bin>