[Oisf-users] Suricata pegs a detect thread and drops packets

[Anoop Saldanha]

Dave,

Nice!  You did manage to catch that thread inside the get() function.
>From the perf-top it indicates that it's the same issue, although it
would be nice if I could look at the contents of the variable "l"
inside that get() function.  Unfortunately from your gdb output it
looks like you forgot to compile libhtp without(i typed it wrongly as
"with" in my previous mail, instead of "without", although I did
specify -g3 later on) optimization.

Can you get the output of "print l" inside the get() function, but
without optimization enabled in the libhtp library.

You will have to rebuild libhtp without optimization.  Go to your
libhtp build directory.  Run the "configure" command like you have
before.  Before you type "make", search for "Makefile" in the libhtp
root/base directory and the "htp" subdirectory, and replace "-g -O2"
with just "-g".  Now run the make command.  Confirm during the build
stage on the console that you don't have "-g -o2" with any of the gcc
commands and instead have just "-g".

You can then do a "make install", OR manually copy the .so files
directory from "htp/.libs/" directory, to replace the libhtp libraries
that you are linking with suricata.

If you still can't see the symbols inside libhtp's functions when you
run suricata, your suricata binary is pointing to the wrong/old libhtp
library.


On Sat, Jun 21, 2014 at 12:42 AM, David Vasil <davidvasil at gmail.com> wrote:
> I was able to do this after Detect5 hit 100% and stayed there.  I reverted
> back to my originally compiled suricata 2.0.1 deb package (without
> --enable-debug) as that flag created a ton of overhead - as you mentioned,
> probably due to not being compiled with optimization - and it also ended up
> core dumping several times.  I copied the unstripped libhtp lib and suricata
> binary (again, without --enable-debug) to the appropriate destinations and
> was able to see the debugging symbols as expected.  Attached is a 'perf top'
> drilling into the annotated code within htp_list_array_get showing where the
> time is being spent (I assume).  9d99, not in the screenshot, shows the
> following:
>
>     0.08 :            9d99:       repz retq
>          :            free(l->elements);
>          :            free(l);
>          :        }
>
> GDB from this is thread here: http://pastebin.com/3tfjTsL0
>
> Thanks!
> -dave
>
>
> On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <anoopsaldanha at gmail.com>
> wrote:
>>
>> I don't think --enable-debug compiles it with optimization.  Instead
>> compile it without optimization, i.e. either -g -O0 or -g -03.  Copy
>> the new binaries over, like you previously did.  You will also have to
>> compile libhtp the same way.  You can either specify this in the
>> environment variable with configure or manually edit the configure
>> script and the makefiles, replacing all "-g -o2" with just "-g".
>>
>> 1. You can start suricata, and wait for one of the detect threads to
>> hit the 100% cpu utilization mark(make a note of the detect
>> threadname).
>> 2. One you see that, attach gdb to the running process, and print the
>> threads using "info threads".  If you see the offending thread stuck
>> in the libhtp get() function call, switch over to that thread using "t
>> <thread_number>" and do a "print l".  The symbol "l" is inside the
>> libhtp get() function call.  Unless you have the detect thread inside
>> the libhtp get() function scope that we are trying to trace, you won't
>> have the symbol available for printing.
>> 3. If when you do a "info threads", you don't see any of the threads
>> currently inside htp get() function(gone out of scope at that instance
>> of time t), continue the process in gdb, and keep a tab on the threads
>> with top/htop, till you see the detect thread(s) again hit the 100%
>> cpu mark, post which you can interrupt the process inside gdb again
>> and hopefully find the detect thread still inside the libhtp get()
>> function context.
>>
>> With the issue at hand, once the thread gets pegged, you should be
>> able to zero-in on the thread pretty quickly.  In case you can't, I'll
>> provide a debug patch to corner the issue.



-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------