[Oisf-users] Suricata pegs a detect thread and drops packets

[Cooper F. Nelson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Do you have transparent huge pages enabled in the kernel?

> $ sudo zcat /proc/config.gz | fgrep CONFIG_TRANSPARENT_HUGEPAGE
> CONFIG_TRANSPARENT_HUGEPAGE=y
> CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y

Before I enabled this I noticed suricata spending lots of time managing
memory.

- -Coop

On 6/20/2014 12:12 PM, David Vasil wrote:
> I was able to do this after Detect5 hit 100% and stayed there.  I
> reverted back to my originally compiled suricata 2.0.1 deb package
> (without --enable-debug) as that flag created a ton of overhead - as you
> mentioned, probably due to not being compiled with optimization - and it
> also ended up core dumping several times.  I copied the unstripped
> libhtp lib and suricata binary (again, without --enable-debug) to the
> appropriate destinations and was able to see the debugging symbols as
> expected.  Attached is a 'perf top' drilling into the annotated code
> within htp_list_array_get showing where the time is being spent (I
> assume).  9d99, not in the screenshot, shows the following:
> 
>     0.08 :            9d99:       repz retq 
>          :            free(l->elements);                 
>          :            free(l);    
>          :        } 
> 
> GDB from this is thread here: http://pastebin.com/3tfjTsL0
> 
> Thanks!
> -dave
> 
> On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <anoopsaldanha at gmail.com
> <mailto:anoopsaldanha at gmail.com>> wrote:
> 
>     I don't think --enable-debug compiles it with optimization.  Instead
>     compile it without optimization, i.e. either -g -O0 or -g -03.  Copy
>     the new binaries over, like you previously did.  You will also have to
>     compile libhtp the same way.  You can either specify this in the
>     environment variable with configure or manually edit the configure
>     script and the makefiles, replacing all "-g -o2" with just "-g".
> 
>     1. You can start suricata, and wait for one of the detect threads to
>     hit the 100% cpu utilization mark(make a note of the detect
>     threadname).
>     2. One you see that, attach gdb to the running process, and print the
>     threads using "info threads".  If you see the offending thread stuck
>     in the libhtp get() function call, switch over to that thread using "t
>     <thread_number>" and do a "print l".  The symbol "l" is inside the
>     libhtp get() function call.  Unless you have the detect thread inside
>     the libhtp get() function scope that we are trying to trace, you won't
>     have the symbol available for printing.
>     3. If when you do a "info threads", you don't see any of the threads
>     currently inside htp get() function(gone out of scope at that instance
>     of time t), continue the process in gdb, and keep a tab on the threads
>     with top/htop, till you see the detect thread(s) again hit the 100%
>     cpu mark, post which you can interrupt the process inside gdb again
>     and hopefully find the detect thread still inside the libhtp get()
>     function context.
> 
>     With the issue at hand, once the thread gets pegged, you should be
>     able to zero-in on the thread pretty quickly.  In case you can't, I'll
>     provide a debug patch to corner the issue.
> 
> 
> 
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
> 


- -- 
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTpIu3AAoJEKIFRYQsa8FWdqAH/2LutdYNjAfdALQcNolfyPl0
WAy8RdQxMvv9yEfeZDV9+NIG8xoVqE0/am/q6HD+YWK62IUsWdQPr/JMDEPv8+Rc
k8cSqlZ75DtBqOqVyGz5R/QwqIJ+TqYygPAxGYpsOM7TtVTQwLZhp00GXkDwobNv
BcnPY9Gu9QKoxAJz+2pKPNNivmCPsDpXIVwFShV/88lGWyhzRxxQnBdyB1Jx7yJR
sQr+2lT5s4hHYMA8EYFiZ7reIf3TFdQsN012tdq7SsB2oPxcC3xZeF2EDCehIozi
ldYQE2fnh1f88Wzb5FpBTLjsqGpmRKLYpBNhqrYUB8LQ2YOHbC5kFZJ2e5ou4Bk=
=M2b7
-----END PGP SIGNATURE-----