[Oisf-users] Suricata pegs a detect thread and drops packets

[David Vasil]

Cooper,
  My kernel (3.2.0.64.76, from the ubuntu precise repo that 12.04.4
SecurityOnion uses) has the following HUGEPAGES options set:

# grep CONFIG_TRANSPARENT_HUGEPAGE /boot/config-3.2.0-64-generic
CONFIG_TRANSPARENT_HUGEPAGE=y
# CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS is not set
CONFIG_TRANSPARENT_HUGEPAGE_MADVISE=y

# cat /sys/kernel/mm/transparent_hugepage/enabled
always [madvise] never

I'll see how it behaves if I set it to 'always' instead of 'madvise'.

-dave


On Fri, Jun 20, 2014 at 2:29 PM, Cooper F. Nelson <cnelson at ucsd.edu> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Do you have transparent huge pages enabled in the kernel?
>
> > $ sudo zcat /proc/config.gz | fgrep CONFIG_TRANSPARENT_HUGEPAGE
> > CONFIG_TRANSPARENT_HUGEPAGE=y
> > CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS=y
>
> Before I enabled this I noticed suricata spending lots of time managing
> memory.
>
> - -Coop
>
> On 6/20/2014 12:12 PM, David Vasil wrote:
> > I was able to do this after Detect5 hit 100% and stayed there.  I
> > reverted back to my originally compiled suricata 2.0.1 deb package
> > (without --enable-debug) as that flag created a ton of overhead - as you
> > mentioned, probably due to not being compiled with optimization - and it
> > also ended up core dumping several times.  I copied the unstripped
> > libhtp lib and suricata binary (again, without --enable-debug) to the
> > appropriate destinations and was able to see the debugging symbols as
> > expected.  Attached is a 'perf top' drilling into the annotated code
> > within htp_list_array_get showing where the time is being spent (I
> > assume).  9d99, not in the screenshot, shows the following:
> >
> >     0.08 :            9d99:       repz retq
> >          :            free(l->elements);
> >          :            free(l);
> >          :        }
> >
> > GDB from this is thread here: http://pastebin.com/3tfjTsL0
> >
> > Thanks!
> > -dave
> >
> > On Fri, Jun 20, 2014 at 9:41 AM, Anoop Saldanha <anoopsaldanha at gmail.com
> > <mailto:anoopsaldanha at gmail.com>> wrote:
> >
> >     I don't think --enable-debug compiles it with optimization.  Instead
> >     compile it without optimization, i.e. either -g -O0 or -g -03.  Copy
> >     the new binaries over, like you previously did.  You will also have
> to
> >     compile libhtp the same way.  You can either specify this in the
> >     environment variable with configure or manually edit the configure
> >     script and the makefiles, replacing all "-g -o2" with just "-g".
> >
> >     1. You can start suricata, and wait for one of the detect threads to
> >     hit the 100% cpu utilization mark(make a note of the detect
> >     threadname).
> >     2. One you see that, attach gdb to the running process, and print the
> >     threads using "info threads".  If you see the offending thread stuck
> >     in the libhtp get() function call, switch over to that thread using
> "t
> >     <thread_number>" and do a "print l".  The symbol "l" is inside the
> >     libhtp get() function call.  Unless you have the detect thread inside
> >     the libhtp get() function scope that we are trying to trace, you
> won't
> >     have the symbol available for printing.
> >     3. If when you do a "info threads", you don't see any of the threads
> >     currently inside htp get() function(gone out of scope at that
> instance
> >     of time t), continue the process in gdb, and keep a tab on the
> threads
> >     with top/htop, till you see the detect thread(s) again hit the 100%
> >     cpu mark, post which you can interrupt the process inside gdb again
> >     and hopefully find the detect thread still inside the libhtp get()
> >     function context.
> >
> >     With the issue at hand, once the thread gets pegged, you should be
> >     able to zero-in on the thread pretty quickly.  In case you can't,
> I'll
> >     provide a debug patch to corner the issue.
> >
> >
> >
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> > List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
> >
>
>
> - --
> Cooper Nelson
> Network Security Analyst
> UCSD ACT Security Team
> cnelson at ucsd.edu x41042
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.17 (MingW32)
> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
> iQEcBAEBAgAGBQJTpIu3AAoJEKIFRYQsa8FWdqAH/2LutdYNjAfdALQcNolfyPl0
> WAy8RdQxMvv9yEfeZDV9+NIG8xoVqE0/am/q6HD+YWK62IUsWdQPr/JMDEPv8+Rc
> k8cSqlZ75DtBqOqVyGz5R/QwqIJ+TqYygPAxGYpsOM7TtVTQwLZhp00GXkDwobNv
> BcnPY9Gu9QKoxAJz+2pKPNNivmCPsDpXIVwFShV/88lGWyhzRxxQnBdyB1Jx7yJR
> sQr+2lT5s4hHYMA8EYFiZ7reIf3TFdQsN012tdq7SsB2oPxcC3xZeF2EDCehIozi
> ldYQE2fnh1f88Wzb5FpBTLjsqGpmRKLYpBNhqrYUB8LQ2YOHbC5kFZJ2e5ou4Bk=
> =M2b7
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140620/4945d892/attachment-0002.html>