<p dir="ltr">Hi Anoop,</p>
<p dir="ltr">Sorry for the late answer. In the last five days Suricata was running, it had 0.28% of packet loss. So definitely, there was a huge improvement.</p>
<p dir="ltr">Now it's quite harder to create the back trace to confirm that the array problem is still causing the drops but I would believe so. </p>
<p dir="ltr">Best regards, <br>
Duarte Silva</p>
<div class="gmail_quote">On 12 Jun 2013 05:24, "Anoop Saldanha" <<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Wed, Jun 5, 2013 at 9:33 PM, Duarte Silva<br>
<<a href="mailto:duarte.silva@serializing.me">duarte.silva@serializing.me</a>> wrote:<br>
> On Wednesday 05 June 2013 16:28:24 Anoop Saldanha wrote:<br>
>> Would it possible to run the current dev master and see how it<br>
>> performs? While the list thing I specified in the previous mail is<br>
>> unchanged in the master, there are a couple of improvements on how we<br>
>> manage the items in the said list, which might make a difference in<br>
>> performance as specified here -<br>
>> <a href="http://www.poona.me/2013/05/suricata-transaction-engine-re-designed.html" target="_blank">http://www.poona.me/2013/05/suricata-transaction-engine-re-designed.html</a><br>
><br>
> I have updated Suricatas to the current development master. We will most<br>
> likely have to wait until tomorrow morning to check if the optimizations<br>
> panned out (it looks promising by the way).<br>
><br>
<br>
See anything with the current master?<br>
<br>
>> On Wed, Jun 5, 2013 at 4:24 PM, Duarte Silva<br>
>><br>
>> <<a href="mailto:duarte.silva@serializing.me">duarte.silva@serializing.me</a>> wrote:<br>
>> > On Wednesday 05 June 2013 16:04:02 Anoop Saldanha wrote:<br>
>> >> On Wed, Jun 5, 2013 at 2:45 PM, Duarte Silva<br>
>> >><br>
>> >> <<a href="mailto:duarte.silva@serializing.me">duarte.silva@serializing.me</a>> wrote:<br>
>> >> > On Thursday 16 May 2013 10:01:26 Duarte Silva wrote:<br>
>> >> >> On Wednesday 15 May 2013 19:54:21 Anoop Saldanha wrote:<br>
>> >> >> > On Wed, May 15, 2013 at 3:55 PM, Duarte Silva<br>
>> >> >> ><br>
>> >> >> > <<a href="mailto:duarte.silva@serializing.me">duarte.silva@serializing.me</a>> wrote:<br>
>> >> >> > > Hi all,<br>
>> >> >> > ><br>
>> >> >> > > I'm currently facing a problem with Suricata. After running for a<br>
>> >> >> > > while,<br>
>> >> >> > > there is always an AF_PACKET thread (workers mode) that hogs the<br>
>> >> >> > > CPU<br>
>> >> >> > > to<br>
>> >> >> > > which it is bound to creating an awful amount of packet loss. I<br>
>> >> >> > > have<br>
>> >> >> > > discarded the><br>
>> >> >> > ><br>
>> >> >> > > following factors:<br>
>> >> >> > > - Number of rules, it has also happened without rules;<br>
>> >> >> > > - Amount of network traffic, I have seen Suricata handle ~18 MBs<br>
>> >> >> > > (150<br>
>> >> >> > > MBps) of><br>
>> >> >> > ><br>
>> >> >> > > traffic without problems with the current configuration and it as<br>
>> >> >> > > also<br>
>> >> >> > > happened with only ~2 MBs of traffic;<br>
>> >> >> > ><br>
>> >> >> > > - Memory, Suricata was only using ~500 MB of it when the CPU<br>
>> >> >> > > usage<br>
>> >> >> > > pegged<br>
>> >> >> > > to><br>
>> >> >> > ><br>
>> >> >> > > 100%;<br>
>> >> >> > ><br>
>> >> >> > > This happens repeatedly and after it happens, Suricata takes a<br>
>> >> >> > > long<br>
>> >> >> > > time<br>
>> >> >> > > to<br>
>> >> >> > > stop. Could some tell me what I can do to debug this issue?<br>
>> >> >> > ><br>
>> >> >> > > Suricata is executed with the following command line:<br>
>> >> >> > ><br>
>> >> >> > > suricata -D -c /etc/suricata/suricata.yaml --pidfile<br>
>> >> >> > > /var/lock/subsys/suricata --af-packet=eth1 --user=suri<br>
>> >> >> > > --group=suri<br>
>> >> >> > ><br>
>> >> >> > > I have also attached any files that can help out in debugging.<br>
>> >> >> ><br>
>> >> >> > While this thread hogs the cpu, can you attach gdb to the suricata<br>
>> >> >> > process, and get a bt for the specified thread, and also all the<br>
>> >> >> > threads.<br>
>> >> >><br>
>> >> >> Follows in the attachments the traces for the hogging thread (I had to<br>
>> >> >> wait<br>
>> >> >> almost height hours for it to happen). I have created three traces in<br>
>> >> >> different times while the AFPacketeth12 was hoging the CPU, all of<br>
>> >> >> them<br>
>> >> >> end<br>
>> >> >> up in the list_array_get in dslib.c.<br>
>> >> >><br>
>> >> >> I will investigate what is happening by looking at the code, when it<br>
>> >> >> happens again I will also take traces for the other threads.<br>
>> >> ><br>
>> >> > Hi,<br>
>> >> ><br>
>> >> > I have taken two more traces when it happened again. Could you please<br>
>> >> > give<br>
>> >> > a little help on this? I think it has something to do with HTTP<br>
>> >> > processing.<br>
>> >><br>
>> >> @Duarte<br>
>> >><br>
>> >> What version of suricata are you running?<br>
>> ><br>
>> > I have updated to the latest and greatest, 1.4.2<br>
>> ><br>
>> >> @Victor.<br>
>> >><br>
>> >> From the last bt that Duarte sent, it looks like the list has grown in<br>
>> >> size. The size is around 4k. Probably that's the reason for the<br>
>> >> slowdown? Every time we inspect state we will end up looping through<br>
>> >> the whole array.<br>
<br>
<br>
<br>
--<br>
-------------------------------<br>
Anoop Saldanha<br>
<a href="http://www.poona.me" target="_blank">http://www.poona.me</a><br>
-------------------------------<br>
</blockquote></div>