[Oisf-users] Help with 99% CPU usage

[Anoop Saldanha]

On Wed, Jun 5, 2013 at 2:45 PM, Duarte Silva
<duarte.silva at serializing.me> wrote:
> On Thursday 16 May 2013 10:01:26 Duarte Silva wrote:
>> On Wednesday 15 May 2013 19:54:21 Anoop Saldanha wrote:
>> > On Wed, May 15, 2013 at 3:55 PM, Duarte Silva
>> >
>> > <duarte.silva at serializing.me> wrote:
>> > > Hi all,
>> > >
>> > > I'm currently facing a problem with Suricata. After running for a while,
>> > > there is always an AF_PACKET thread (workers mode) that hogs the CPU to
>> > > which it is bound to creating an awful amount of packet loss. I have
>> > > discarded the>
>> > >
>> > > following factors:
>> > >  - Number of rules, it has also happened without rules;
>> > >  - Amount of network traffic, I have seen Suricata handle ~18 MBs (150
>> > >  MBps) of>
>> > >
>> > > traffic without problems with the current configuration and it as also
>> > > happened with only ~2 MBs of traffic;
>> > >
>> > >  - Memory, Suricata was only using ~500 MB of it when the CPU usage
>> > >  pegged
>> > >  to>
>> > >
>> > > 100%;
>> > >
>> > > This happens repeatedly and after it happens, Suricata takes a long time
>> > > to
>> > > stop. Could some tell me what I can do to debug this issue?
>> > >
>> > > Suricata is executed with the following command line:
>> > >
>> > > suricata -D -c /etc/suricata/suricata.yaml --pidfile
>> > > /var/lock/subsys/suricata --af-packet=eth1 --user=suri --group=suri
>> > >
>> > > I have also attached any files that can help out in debugging.
>> >
>> > While this thread hogs the cpu, can you attach gdb to the suricata
>> > process, and get a bt for the specified thread, and also all the
>> > threads.
>>
>> Follows in the attachments the traces for the hogging thread (I had to wait
>> almost height hours for it to happen). I have created three traces in
>> different times while the AFPacketeth12 was hoging the CPU, all of them end
>> up in the list_array_get in dslib.c.
>>
>> I will investigate what is happening by looking at the code, when it happens
>> again I will also take traces for the other threads.
>
> Hi,
>
> I have taken two more traces when it happened again. Could you please give a
> little help on this? I think it has something to do with HTTP processing.
>

@Duarte

What version of suricata are you running?

@Victor.

>From the last bt that Duarte sent, it looks like the list has grown in
size.  The size is around 4k.  Probably that's the reason for the
slowdown?  Every time we inspect state we will end up looping through
the whole array.


-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------