[Oisf-users] Suricata pegs a detect thread and drops packets

[David Vasil]

I was able to do this after Detect5 hit 100% and stayed there.  I reverted
back to my originally compiled suricata 2.0.1 deb package (without
--enable-debug) as that flag created a ton of overhead - as you mentioned,
probably due to not being compiled with optimization - and it also ended up
core dumping several times.  I copied the unstripped libhtp lib and
suricata binary (again, without --enable-debug) to the appropriate
destinations and was able to see the debugging symbols as expected.
 Attached is a 'perf top' drilling into the annotated code within
htp_list_array_get showing where the time is being spent (I assume).  9d99,
not in the screenshot, shows the following:

    0.08 :            9d99:       repz retq
         :            free(l->elements);
         :            free(l);
         :        }

GDB from this is thread here: http://pastebin.com/3tfjTsL0

Thanks!
-dave

On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <anoopsaldanha at gmail.com>
wrote:

> I don't think --enable-debug compiles it with optimization.  Instead
> compile it without optimization, i.e. either -g -O0 or -g -03.  Copy
> the new binaries over, like you previously did.  You will also have to
> compile libhtp the same way.  You can either specify this in the
> environment variable with configure or manually edit the configure
> script and the makefiles, replacing all "-g -o2" with just "-g".
>
> 1. You can start suricata, and wait for one of the detect threads to
> hit the 100% cpu utilization mark(make a note of the detect
> threadname).
> 2. One you see that, attach gdb to the running process, and print the
> threads using "info threads".  If you see the offending thread stuck
> in the libhtp get() function call, switch over to that thread using "t
> <thread_number>" and do a "print l".  The symbol "l" is inside the
> libhtp get() function call.  Unless you have the detect thread inside
> the libhtp get() function scope that we are trying to trace, you won't
> have the symbol available for printing.
> 3. If when you do a "info threads", you don't see any of the threads
> currently inside htp get() function(gone out of scope at that instance
> of time t), continue the process in gdb, and keep a tab on the threads
> with top/htop, till you see the detect thread(s) again hit the 100%
> cpu mark, post which you can interrupt the process inside gdb again
> and hopefully find the detect thread still inside the libhtp get()
> function context.
>
> With the issue at hand, once the thread gets pegged, you should be
> able to zero-in on the thread pretty quickly.  In case you can't, I'll
> provide a debug patch to corner the issue.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140620/7512493a/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: htp_list_array_get-perf_top_annotate.png
Type: image/png
Size: 230013 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140620/7512493a/attachment-0002.png>