[Oisf-users] Help with 99% CPU usage

[Anoop Saldanha]

Duarte,

Good to know that the performance has improved.

Tracking the list issue here -
https://redmine.openinfosecfoundation.org/issues/822, although it's
unlikely the drop is due to the the list.

The setup might need some fine-tuning for the drops.  Probably
requires a mail of its own

On Wed, Jun 12, 2013 at 11:51 AM, Duarte Silva
<duarte.silva at serializing.me> wrote:
> Hi Anoop,
>
> Sorry for the late answer. In the last five days Suricata was running, it
> had 0.28% of packet loss. So definitely, there was a huge improvement.
>
> Now it's quite harder to create the back trace to confirm that the array
> problem is still causing the drops but I would believe so.
>
> Best regards,
> Duarte Silva
>
> On 12 Jun 2013 05:24, "Anoop Saldanha" <anoopsaldanha at gmail.com> wrote:
>>
>> On Wed, Jun 5, 2013 at 9:33 PM, Duarte Silva
>> <duarte.silva at serializing.me> wrote:
>> > On Wednesday 05 June 2013 16:28:24 Anoop Saldanha wrote:
>> >> Would it possible to run the current dev master and see how it
>> >> performs?  While the list thing I specified in the previous mail is
>> >> unchanged in the master, there are a couple of improvements on how we
>> >> manage the items in the said list, which might make a difference in
>> >> performance as specified here -
>> >>
>> >> http://www.poona.me/2013/05/suricata-transaction-engine-re-designed.html
>> >
>> > I have updated Suricatas to the current development master. We will most
>> > likely have to wait until tomorrow morning to check if the optimizations
>> > panned out (it looks promising by the way).
>> >
>>
>> See anything with the current master?
>>
>> >> On Wed, Jun 5, 2013 at 4:24 PM, Duarte Silva
>> >>
>> >> <duarte.silva at serializing.me> wrote:
>> >> > On Wednesday 05 June 2013 16:04:02 Anoop Saldanha wrote:
>> >> >> On Wed, Jun 5, 2013 at 2:45 PM, Duarte Silva
>> >> >>
>> >> >> <duarte.silva at serializing.me> wrote:
>> >> >> > On Thursday 16 May 2013 10:01:26 Duarte Silva wrote:
>> >> >> >> On Wednesday 15 May 2013 19:54:21 Anoop Saldanha wrote:
>> >> >> >> > On Wed, May 15, 2013 at 3:55 PM, Duarte Silva
>> >> >> >> >
>> >> >> >> > <duarte.silva at serializing.me> wrote:
>> >> >> >> > > Hi all,
>> >> >> >> > >
>> >> >> >> > > I'm currently facing a problem with Suricata. After running
>> >> >> >> > > for a
>> >> >> >> > > while,
>> >> >> >> > > there is always an AF_PACKET thread (workers mode) that hogs
>> >> >> >> > > the
>> >> >> >> > > CPU
>> >> >> >> > > to
>> >> >> >> > > which it is bound to creating an awful amount of packet loss.
>> >> >> >> > > I
>> >> >> >> > > have
>> >> >> >> > > discarded the>
>> >> >> >> > >
>> >> >> >> > > following factors:
>> >> >> >> > >  - Number of rules, it has also happened without rules;
>> >> >> >> > >  - Amount of network traffic, I have seen Suricata handle ~18
>> >> >> >> > > MBs
>> >> >> >> > >  (150
>> >> >> >> > >  MBps) of>
>> >> >> >> > >
>> >> >> >> > > traffic without problems with the current configuration and
>> >> >> >> > > it as
>> >> >> >> > > also
>> >> >> >> > > happened with only ~2 MBs of traffic;
>> >> >> >> > >
>> >> >> >> > >  - Memory, Suricata was only using ~500 MB of it when the CPU
>> >> >> >> > >  usage
>> >> >> >> > >  pegged
>> >> >> >> > >  to>
>> >> >> >> > >
>> >> >> >> > > 100%;
>> >> >> >> > >
>> >> >> >> > > This happens repeatedly and after it happens, Suricata takes
>> >> >> >> > > a
>> >> >> >> > > long
>> >> >> >> > > time
>> >> >> >> > > to
>> >> >> >> > > stop. Could some tell me what I can do to debug this issue?
>> >> >> >> > >
>> >> >> >> > > Suricata is executed with the following command line:
>> >> >> >> > >
>> >> >> >> > > suricata -D -c /etc/suricata/suricata.yaml --pidfile
>> >> >> >> > > /var/lock/subsys/suricata --af-packet=eth1 --user=suri
>> >> >> >> > > --group=suri
>> >> >> >> > >
>> >> >> >> > > I have also attached any files that can help out in
>> >> >> >> > > debugging.
>> >> >> >> >
>> >> >> >> > While this thread hogs the cpu, can you attach gdb to the
>> >> >> >> > suricata
>> >> >> >> > process, and get a bt for the specified thread, and also all
>> >> >> >> > the
>> >> >> >> > threads.
>> >> >> >>
>> >> >> >> Follows in the attachments the traces for the hogging thread (I
>> >> >> >> had to
>> >> >> >> wait
>> >> >> >> almost height hours for it to happen). I have created three
>> >> >> >> traces in
>> >> >> >> different times while the AFPacketeth12 was hoging the CPU, all
>> >> >> >> of
>> >> >> >> them
>> >> >> >> end
>> >> >> >> up in the list_array_get in dslib.c.
>> >> >> >>
>> >> >> >> I will investigate what is happening by looking at the code, when
>> >> >> >> it
>> >> >> >> happens again I will also take traces for the other threads.
>> >> >> >
>> >> >> > Hi,
>> >> >> >
>> >> >> > I have taken two more traces when it happened again. Could you
>> >> >> > please
>> >> >> > give
>> >> >> > a little help on this? I think it has something to do with HTTP
>> >> >> > processing.
>> >> >>
>> >> >> @Duarte
>> >> >>
>> >> >> What version of suricata are you running?
>> >> >
>> >> > I have updated to the latest and greatest, 1.4.2
>> >> >
>> >> >> @Victor.
>> >> >>
>> >> >> From the last bt that Duarte sent, it looks like the list has grown
>> >> >> in
>> >> >> size.  The size is around 4k.  Probably that's the reason for the
>> >> >> slowdown?  Every time we inspect state we will end up looping
>> >> >> through
>> >> >> the whole array.
>>
>>
>>
>> --
>> -------------------------------
>> Anoop Saldanha
>> http://www.poona.me
>> -------------------------------



-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------