<div dir="ltr">Here is the result after removing all of the -O2's from the libhtp makefile and waiting for a detect thread to hit 100%.<div><br></div><div><a href="http://pastebin.com/aHp415HV">http://pastebin.com/aHp415HV</a><br>
</div><div><br></div><div>Also, setting transparent hugepages to always did not seem to prevent this from occurring.</div><div><br></div><div>-dave</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 20, 2014 at 10:36 PM, Anoop Saldanha <span dir="ltr"><<a href="mailto:anoopsaldanha@gmail.com" target="_blank">anoopsaldanha@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Dave,<br>
<br>
Nice! You did manage to catch that thread inside the get() function.<br>
>From the perf-top it indicates that it's the same issue, although it<br>
would be nice if I could look at the contents of the variable "l"<br>
inside that get() function. Unfortunately from your gdb output it<br>
looks like you forgot to compile libhtp without(i typed it wrongly as<br>
"with" in my previous mail, instead of "without", although I did<br>
specify -g3 later on) optimization.<br>
<br>
Can you get the output of "print l" inside the get() function, but<br>
without optimization enabled in the libhtp library.<br>
<br>
You will have to rebuild libhtp without optimization. Go to your<br>
libhtp build directory. Run the "configure" command like you have<br>
before. Before you type "make", search for "Makefile" in the libhtp<br>
root/base directory and the "htp" subdirectory, and replace "-g -O2"<br>
with just "-g". Now run the make command. Confirm during the build<br>
stage on the console that you don't have "-g -o2" with any of the gcc<br>
commands and instead have just "-g".<br>
<br>
You can then do a "make install", OR manually copy the .so files<br>
directory from "htp/.libs/" directory, to replace the libhtp libraries<br>
that you are linking with suricata.<br>
<br>
If you still can't see the symbols inside libhtp's functions when you<br>
run suricata, your suricata binary is pointing to the wrong/old libhtp<br>
library.<br>
<div class="HOEnZb"><div class="h5"><br>
<br>
On Sat, Jun 21, 2014 at 12:42 AM, David Vasil <<a href="mailto:davidvasil@gmail.com">davidvasil@gmail.com</a>> wrote:<br>
> I was able to do this after Detect5 hit 100% and stayed there. I reverted<br>
> back to my originally compiled suricata 2.0.1 deb package (without<br>
> --enable-debug) as that flag created a ton of overhead - as you mentioned,<br>
> probably due to not being compiled with optimization - and it also ended up<br>
> core dumping several times. I copied the unstripped libhtp lib and suricata<br>
> binary (again, without --enable-debug) to the appropriate destinations and<br>
> was able to see the debugging symbols as expected. Attached is a 'perf top'<br>
> drilling into the annotated code within htp_list_array_get showing where the<br>
> time is being spent (I assume). 9d99, not in the screenshot, shows the<br>
> following:<br>
><br>
> 0.08 : 9d99: repz retq<br>
> : free(l->elements);<br>
> : free(l);<br>
> : }<br>
><br>
> GDB from this is thread here: <a href="http://pastebin.com/3tfjTsL0" target="_blank">http://pastebin.com/3tfjTsL0</a><br>
><br>
> Thanks!<br>
> -dave<br>
><br>
><br>
> On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a>><br>
> wrote:<br>
>><br>
>> I don't think --enable-debug compiles it with optimization. Instead<br>
>> compile it without optimization, i.e. either -g -O0 or -g -03. Copy<br>
>> the new binaries over, like you previously did. You will also have to<br>
>> compile libhtp the same way. You can either specify this in the<br>
>> environment variable with configure or manually edit the configure<br>
>> script and the makefiles, replacing all "-g -o2" with just "-g".<br>
>><br>
>> 1. You can start suricata, and wait for one of the detect threads to<br>
>> hit the 100% cpu utilization mark(make a note of the detect<br>
>> threadname).<br>
>> 2. One you see that, attach gdb to the running process, and print the<br>
>> threads using "info threads". If you see the offending thread stuck<br>
>> in the libhtp get() function call, switch over to that thread using "t<br>
>> <thread_number>" and do a "print l". The symbol "l" is inside the<br>
>> libhtp get() function call. Unless you have the detect thread inside<br>
>> the libhtp get() function scope that we are trying to trace, you won't<br>
>> have the symbol available for printing.<br>
>> 3. If when you do a "info threads", you don't see any of the threads<br>
>> currently inside htp get() function(gone out of scope at that instance<br>
>> of time t), continue the process in gdb, and keep a tab on the threads<br>
>> with top/htop, till you see the detect thread(s) again hit the 100%<br>
>> cpu mark, post which you can interrupt the process inside gdb again<br>
>> and hopefully find the detect thread still inside the libhtp get()<br>
>> function context.<br>
>><br>
>> With the issue at hand, once the thread gets pegged, you should be<br>
>> able to zero-in on the thread pretty quickly. In case you can't, I'll<br>
>> provide a debug patch to corner the issue.<br>
<br>
<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">--<br>
-------------------------------<br>
Anoop Saldanha<br>
<a href="http://www.poona.me" target="_blank">http://www.poona.me</a><br>
-------------------------------<br>
</div></div></blockquote></div><br></div>