[Oisf-users] Suricata pegs a detect thread and drops packets

[David Vasil]

Here is the result after removing all of the -O2's from the libhtp makefile
and waiting for a detect thread to hit 100%.

http://pastebin.com/aHp415HV

Also, setting transparent hugepages to always did not seem to prevent this
from occurring.

-dave


On Fri, Jun 20, 2014 at 10:36 PM, Anoop Saldanha <anoopsaldanha at gmail.com>
wrote:

> Dave,
>
> Nice!  You did manage to catch that thread inside the get() function.
> From the perf-top it indicates that it's the same issue, although it
> would be nice if I could look at the contents of the variable "l"
> inside that get() function.  Unfortunately from your gdb output it
> looks like you forgot to compile libhtp without(i typed it wrongly as
> "with" in my previous mail, instead of "without", although I did
> specify -g3 later on) optimization.
>
> Can you get the output of "print l" inside the get() function, but
> without optimization enabled in the libhtp library.
>
> You will have to rebuild libhtp without optimization.  Go to your
> libhtp build directory.  Run the "configure" command like you have
> before.  Before you type "make", search for "Makefile" in the libhtp
> root/base directory and the "htp" subdirectory, and replace "-g -O2"
> with just "-g".  Now run the make command.  Confirm during the build
> stage on the console that you don't have "-g -o2" with any of the gcc
> commands and instead have just "-g".
>
> You can then do a "make install", OR manually copy the .so files
> directory from "htp/.libs/" directory, to replace the libhtp libraries
> that you are linking with suricata.
>
> If you still can't see the symbols inside libhtp's functions when you
> run suricata, your suricata binary is pointing to the wrong/old libhtp
> library.
>
>
> On Sat, Jun 21, 2014 at 12:42 AM, David Vasil <davidvasil at gmail.com>
> wrote:
> > I was able to do this after Detect5 hit 100% and stayed there.  I
> reverted
> > back to my originally compiled suricata 2.0.1 deb package (without
> > --enable-debug) as that flag created a ton of overhead - as you
> mentioned,
> > probably due to not being compiled with optimization - and it also ended
> up
> > core dumping several times.  I copied the unstripped libhtp lib and
> suricata
> > binary (again, without --enable-debug) to the appropriate destinations
> and
> > was able to see the debugging symbols as expected.  Attached is a 'perf
> top'
> > drilling into the annotated code within htp_list_array_get showing where
> the
> > time is being spent (I assume).  9d99, not in the screenshot, shows the
> > following:
> >
> >     0.08 :            9d99:       repz retq
> >          :            free(l->elements);
> >          :            free(l);
> >          :        }
> >
> > GDB from this is thread here: http://pastebin.com/3tfjTsL0
> >
> > Thanks!
> > -dave
> >
> >
> > On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <anoopsaldanha at gmail.com
> >
> > wrote:
> >>
> >> I don't think --enable-debug compiles it with optimization.  Instead
> >> compile it without optimization, i.e. either -g -O0 or -g -03.  Copy
> >> the new binaries over, like you previously did.  You will also have to
> >> compile libhtp the same way.  You can either specify this in the
> >> environment variable with configure or manually edit the configure
> >> script and the makefiles, replacing all "-g -o2" with just "-g".
> >>
> >> 1. You can start suricata, and wait for one of the detect threads to
> >> hit the 100% cpu utilization mark(make a note of the detect
> >> threadname).
> >> 2. One you see that, attach gdb to the running process, and print the
> >> threads using "info threads".  If you see the offending thread stuck
> >> in the libhtp get() function call, switch over to that thread using "t
> >> <thread_number>" and do a "print l".  The symbol "l" is inside the
> >> libhtp get() function call.  Unless you have the detect thread inside
> >> the libhtp get() function scope that we are trying to trace, you won't
> >> have the symbol available for printing.
> >> 3. If when you do a "info threads", you don't see any of the threads
> >> currently inside htp get() function(gone out of scope at that instance
> >> of time t), continue the process in gdb, and keep a tab on the threads
> >> with top/htop, till you see the detect thread(s) again hit the 100%
> >> cpu mark, post which you can interrupt the process inside gdb again
> >> and hopefully find the detect thread still inside the libhtp get()
> >> function context.
> >>
> >> With the issue at hand, once the thread gets pegged, you should be
> >> able to zero-in on the thread pretty quickly.  In case you can't, I'll
> >> provide a debug patch to corner the issue.
>
>
>
> --
> -------------------------------
> Anoop Saldanha
> http://www.poona.me
> -------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140623/890d6037/attachment-0002.html>