[Oisf-users] Help with 99% CPU usage

[Duarte Silva]

On Wednesday 05 June 2013 16:28:24 Anoop Saldanha wrote:
> Would it possible to run the current dev master and see how it
> performs?  While the list thing I specified in the previous mail is
> unchanged in the master, there are a couple of improvements on how we
> manage the items in the said list, which might make a difference in
> performance as specified here -
> http://www.poona.me/2013/05/suricata-transaction-engine-re-designed.html

I have updated Suricatas to the current development master. We will most 
likely have to wait until tomorrow morning to check if the optimizations 
panned out (it looks promising by the way).

> On Wed, Jun 5, 2013 at 4:24 PM, Duarte Silva
> 
> <duarte.silva at serializing.me> wrote:
> > On Wednesday 05 June 2013 16:04:02 Anoop Saldanha wrote:
> >> On Wed, Jun 5, 2013 at 2:45 PM, Duarte Silva
> >> 
> >> <duarte.silva at serializing.me> wrote:
> >> > On Thursday 16 May 2013 10:01:26 Duarte Silva wrote:
> >> >> On Wednesday 15 May 2013 19:54:21 Anoop Saldanha wrote:
> >> >> > On Wed, May 15, 2013 at 3:55 PM, Duarte Silva
> >> >> > 
> >> >> > <duarte.silva at serializing.me> wrote:
> >> >> > > Hi all,
> >> >> > > 
> >> >> > > I'm currently facing a problem with Suricata. After running for a
> >> >> > > while,
> >> >> > > there is always an AF_PACKET thread (workers mode) that hogs the
> >> >> > > CPU
> >> >> > > to
> >> >> > > which it is bound to creating an awful amount of packet loss. I
> >> >> > > have
> >> >> > > discarded the>
> >> >> > > 
> >> >> > > following factors:
> >> >> > >  - Number of rules, it has also happened without rules;
> >> >> > >  - Amount of network traffic, I have seen Suricata handle ~18 MBs
> >> >> > >  (150
> >> >> > >  MBps) of>
> >> >> > > 
> >> >> > > traffic without problems with the current configuration and it as
> >> >> > > also
> >> >> > > happened with only ~2 MBs of traffic;
> >> >> > > 
> >> >> > >  - Memory, Suricata was only using ~500 MB of it when the CPU
> >> >> > >  usage
> >> >> > >  pegged
> >> >> > >  to>
> >> >> > > 
> >> >> > > 100%;
> >> >> > > 
> >> >> > > This happens repeatedly and after it happens, Suricata takes a
> >> >> > > long
> >> >> > > time
> >> >> > > to
> >> >> > > stop. Could some tell me what I can do to debug this issue?
> >> >> > > 
> >> >> > > Suricata is executed with the following command line:
> >> >> > > 
> >> >> > > suricata -D -c /etc/suricata/suricata.yaml --pidfile
> >> >> > > /var/lock/subsys/suricata --af-packet=eth1 --user=suri
> >> >> > > --group=suri
> >> >> > > 
> >> >> > > I have also attached any files that can help out in debugging.
> >> >> > 
> >> >> > While this thread hogs the cpu, can you attach gdb to the suricata
> >> >> > process, and get a bt for the specified thread, and also all the
> >> >> > threads.
> >> >> 
> >> >> Follows in the attachments the traces for the hogging thread (I had to
> >> >> wait
> >> >> almost height hours for it to happen). I have created three traces in
> >> >> different times while the AFPacketeth12 was hoging the CPU, all of
> >> >> them
> >> >> end
> >> >> up in the list_array_get in dslib.c.
> >> >> 
> >> >> I will investigate what is happening by looking at the code, when it
> >> >> happens again I will also take traces for the other threads.
> >> > 
> >> > Hi,
> >> > 
> >> > I have taken two more traces when it happened again. Could you please
> >> > give
> >> > a little help on this? I think it has something to do with HTTP
> >> > processing.
> >> 
> >> @Duarte
> >> 
> >> What version of suricata are you running?
> > 
> > I have updated to the latest and greatest, 1.4.2
> > 
> >> @Victor.
> >> 
> >> From the last bt that Duarte sent, it looks like the list has grown in
> >> size.  The size is around 4k.  Probably that's the reason for the
> >> slowdown?  Every time we inspect state we will end up looping through
> >> the whole array.