[Oisf-users] Help with 99% CPU usage

[Anoop Saldanha]

Would it possible to run the current dev master and see how it
performs?  While the list thing I specified in the previous mail is
unchanged in the master, there are a couple of improvements on how we
manage the items in the said list, which might make a difference in
performance as specified here -
http://www.poona.me/2013/05/suricata-transaction-engine-re-designed.html

On Wed, Jun 5, 2013 at 4:24 PM, Duarte Silva
<duarte.silva at serializing.me> wrote:
> On Wednesday 05 June 2013 16:04:02 Anoop Saldanha wrote:
>> On Wed, Jun 5, 2013 at 2:45 PM, Duarte Silva
>>
>> <duarte.silva at serializing.me> wrote:
>> > On Thursday 16 May 2013 10:01:26 Duarte Silva wrote:
>> >> On Wednesday 15 May 2013 19:54:21 Anoop Saldanha wrote:
>> >> > On Wed, May 15, 2013 at 3:55 PM, Duarte Silva
>> >> >
>> >> > <duarte.silva at serializing.me> wrote:
>> >> > > Hi all,
>> >> > >
>> >> > > I'm currently facing a problem with Suricata. After running for a
>> >> > > while,
>> >> > > there is always an AF_PACKET thread (workers mode) that hogs the CPU
>> >> > > to
>> >> > > which it is bound to creating an awful amount of packet loss. I have
>> >> > > discarded the>
>> >> > >
>> >> > > following factors:
>> >> > >  - Number of rules, it has also happened without rules;
>> >> > >  - Amount of network traffic, I have seen Suricata handle ~18 MBs
>> >> > >  (150
>> >> > >  MBps) of>
>> >> > >
>> >> > > traffic without problems with the current configuration and it as
>> >> > > also
>> >> > > happened with only ~2 MBs of traffic;
>> >> > >
>> >> > >  - Memory, Suricata was only using ~500 MB of it when the CPU usage
>> >> > >  pegged
>> >> > >  to>
>> >> > >
>> >> > > 100%;
>> >> > >
>> >> > > This happens repeatedly and after it happens, Suricata takes a long
>> >> > > time
>> >> > > to
>> >> > > stop. Could some tell me what I can do to debug this issue?
>> >> > >
>> >> > > Suricata is executed with the following command line:
>> >> > >
>> >> > > suricata -D -c /etc/suricata/suricata.yaml --pidfile
>> >> > > /var/lock/subsys/suricata --af-packet=eth1 --user=suri --group=suri
>> >> > >
>> >> > > I have also attached any files that can help out in debugging.
>> >> >
>> >> > While this thread hogs the cpu, can you attach gdb to the suricata
>> >> > process, and get a bt for the specified thread, and also all the
>> >> > threads.
>> >>
>> >> Follows in the attachments the traces for the hogging thread (I had to
>> >> wait
>> >> almost height hours for it to happen). I have created three traces in
>> >> different times while the AFPacketeth12 was hoging the CPU, all of them
>> >> end
>> >> up in the list_array_get in dslib.c.
>> >>
>> >> I will investigate what is happening by looking at the code, when it
>> >> happens again I will also take traces for the other threads.
>> >
>> > Hi,
>> >
>> > I have taken two more traces when it happened again. Could you please give
>> > a little help on this? I think it has something to do with HTTP
>> > processing.
>> @Duarte
>>
>> What version of suricata are you running?
>
> I have updated to the latest and greatest, 1.4.2
>
>>
>> @Victor.
>>
>> From the last bt that Duarte sent, it looks like the list has grown in
>> size.  The size is around 4k.  Probably that's the reason for the
>> slowdown?  Every time we inspect state we will end up looping through
>> the whole array.



-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------