<div dir="ltr">Cooper,<div> My kernel (3.2.0.64.76, from the ubuntu precise repo that 12.04.4 SecurityOnion uses) has the following HUGEPAGES options set:</div><div><br></div><div><div># grep CONFIG_TRANSPARENT_HUGEPAGE /boot/config-3.2.0-64-generic </div>
<div>CONFIG_TRANSPARENT_HUGEPAGE=y</div><div># CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set</div><div>CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y</div></div><div><br></div><div><div># cat /sys/kernel/mm/transparent_hugepage/enabled</div>
<div>always [madvise] never</div></div><div><br></div><div>I'll see how it behaves if I set it to 'always' instead of 'madvise'.</div><div><br></div><div>-dave</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jun 20, 2014 at 2:29 PM, Cooper F. Nelson <span dir="ltr"><<a href="mailto:cnelson@ucsd.edu" target="_blank">cnelson@ucsd.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">-----BEGIN PGP SIGNED MESSAGE-----<br>
Hash: SHA1<br>
<br>
Do you have transparent huge pages enabled in the kernel?<br>
<br>
> $ sudo zcat /proc/config.gz | fgrep CONFIG_TRANSPARENT_HUGEPAGE<br>
> CONFIG_TRANSPARENT_HUGEPAGE=y<br>
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y<br>
<br>
Before I enabled this I noticed suricata spending lots of time managing<br>
memory.<br>
<br>
- -Coop<br>
<div class=""><br>
On 6/20/2014 12:12 PM, David Vasil wrote:<br>
> I was able to do this after Detect5 hit 100% and stayed there. I<br>
> reverted back to my originally compiled suricata 2.0.1 deb package<br>
> (without --enable-debug) as that flag created a ton of overhead - as you<br>
> mentioned, probably due to not being compiled with optimization - and it<br>
> also ended up core dumping several times. I copied the unstripped<br>
> libhtp lib and suricata binary (again, without --enable-debug) to the<br>
> appropriate destinations and was able to see the debugging symbols as<br>
> expected. Attached is a 'perf top' drilling into the annotated code<br>
> within htp_list_array_get showing where the time is being spent (I<br>
> assume). 9d99, not in the screenshot, shows the following:<br>
><br>
> 0.08 : 9d99: repz retq<br>
> : free(l->elements);<br>
> : free(l);<br>
> : }<br>
><br>
> GDB from this is thread here: <a href="http://pastebin.com/3tfjTsL0" target="_blank">http://pastebin.com/3tfjTsL0</a><br>
><br>
> Thanks!<br>
> -dave<br>
><br>
> On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a><br>
</div><div><div class="h5">> <mailto:<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a>>> wrote:<br>
><br>
> I don't think --enable-debug compiles it with optimization. Instead<br>
> compile it without optimization, i.e. either -g -O0 or -g -03. Copy<br>
> the new binaries over, like you previously did. You will also have to<br>
> compile libhtp the same way. You can either specify this in the<br>
> environment variable with configure or manually edit the configure<br>
> script and the makefiles, replacing all "-g -o2" with just "-g".<br>
><br>
> 1. You can start suricata, and wait for one of the detect threads to<br>
> hit the 100% cpu utilization mark(make a note of the detect<br>
> threadname).<br>
> 2. One you see that, attach gdb to the running process, and print the<br>
> threads using "info threads". If you see the offending thread stuck<br>
> in the libhtp get() function call, switch over to that thread using "t<br>
> <thread_number>" and do a "print l". The symbol "l" is inside the<br>
> libhtp get() function call. Unless you have the detect thread inside<br>
> the libhtp get() function scope that we are trying to trace, you won't<br>
> have the symbol available for printing.<br>
> 3. If when you do a "info threads", you don't see any of the threads<br>
> currently inside htp get() function(gone out of scope at that instance<br>
> of time t), continue the process in gdb, and keep a tab on the threads<br>
> with top/htop, till you see the detect thread(s) again hit the 100%<br>
> cpu mark, post which you can interrupt the process inside gdb again<br>
> and hopefully find the detect thread still inside the libhtp get()<br>
> function context.<br>
><br>
> With the issue at hand, once the thread gets pegged, you should be<br>
> able to zero-in on the thread pretty quickly. In case you can't, I'll<br>
> provide a debug patch to corner the issue.<br>
><br>
><br>
><br>
</div></div><div class="">> _______________________________________________<br>
> Suricata IDS Users mailing list: <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a><br>
> Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Support: <a href="http://suricata-ids.org/support/" target="_blank">http://suricata-ids.org/support/</a><br>
> List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
> OISF: <a href="http://www.openinfosecfoundation.org/" target="_blank">http://www.openinfosecfoundation.org/</a><br>
><br>
<br>
<br>
</div>- --<br>
Cooper Nelson<br>
Network Security Analyst<br>
UCSD ACT Security Team<br>
<a href="mailto:cnelson@ucsd.edu">cnelson@ucsd.edu</a> x41042<br>
-----BEGIN PGP SIGNATURE-----<br>
Version: GnuPG v2.0.17 (MingW32)<br>
Comment: Using GnuPG with Thunderbird - <a href="http://www.enigmail.net/" target="_blank">http://www.enigmail.net/</a><br>
<br>
iQEcBAEBAgAGBQJTpIu3AAoJEKIFRYQsa8FWdqAH/2LutdYNjAfdALQcNolfyPl0<br>
WAy8RdQxMvv9yEfeZDV9+NIG8xoVqE0/am/q6HD+YWK62IUsWdQPr/JMDEPv8+Rc<br>
k8cSqlZ75DtBqOqVyGz5R/QwqIJ+TqYygPAxGYpsOM7TtVTQwLZhp00GXkDwobNv<br>
BcnPY9Gu9QKoxAJz+2pKPNNivmCPsDpXIVwFShV/88lGWyhzRxxQnBdyB1Jx7yJR<br>
sQr+2lT5s4hHYMA8EYFiZ7reIf3TFdQsN012tdq7SsB2oPxcC3xZeF2EDCehIozi<br>
ldYQE2fnh1f88Wzb5FpBTLjsqGpmRKLYpBNhqrYUB8LQ2YOHbC5kFZJ2e5ou4Bk=<br>
=M2b7<br>
-----END PGP SIGNATURE-----<br>
</blockquote></div><br></div></div>