[Oisf-devel] Suricata dies (core dump) w/ multiple NICs

Sat May 23 23:11:52 UTC 2015

On Thu, May 21, 2015 at 6:18 PM, Peter Manev <petermanev at gmail.com> wrote:

> On Thu, May 21, 2015 at 11:12 PM, Eduardo Meyer <dudu.meyer at gmail.com>
> wrote:
> > On Thu, May 21, 2015 at 6:10 AM, Peter Manev <petermanev at gmail.com>
> wrote:
> >>
> >> On Thu, May 21, 2015 at 1:02 AM, Eduardo Meyer <dudu.meyer at gmail.com>
> >> wrote:
> >> > Hello,
> >> >
> >> > I am running Suricata 2.0.8 RELEASE with 3 interfaces, and from times
> to
> >> > times suricata simply dies. This is the process arguments in use:
> >> >
> >> > root         45492   1.0  1.5 1299164 251564  -  Is    4:20PM
> >> > 84:38.13
> >> > /usr/local/bin/suricata -D -i bridge1 -i bridge2 -i bridge0 --pidfile
> >> > /var/run/suricata_bridge0.pid -c /usr/local/etc/suricata/suricata.yaml
> >> >
> >> > I could not find a pattern when Suricata dies. Sometimes it's a high
> >> > pps/memory/bandwidth usage profile, sometimes it's a low demand hour
> >> > with
> >> > just a couple pps passing the suricata system.
> >> >
> >> > It never dies with a single interface. It dies for bridged ports,
> >> > trunked
> >> > ports as well as for physical untagged ports, so it does not seem to
> be
> >> > related to virtual or real NICs it's listening at, although I noticed
> it
> >> > dies more frequently on bridged interfaces like the above scenario.
> >> >
> >> > Is there anything I should look at with special attention on
> >> > suricata.yaml?
> >> >
> >> > I have a suricata.core everytime it dies. How can I produce useful
> >> > information from it?
> >>
> >> If you have a core dump and can reproduce the issue consistently - you
> >> can have a look at this guide here -
> >>
> >>
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs
> >> how to extract useful info. Then you can open a bug report should you
> >> consider.
> >>
> >> Thank you
> >
> >
> >
> > What else should I do to the bug the cause? I am no gdb familiarized in
> any
> > ways, so I can't move forth, so far this is what I had only:
> >
> > gdb /usr/local/bin/suricata /suricata.core
> > GNU gdb 6.1.1 [FreeBSD]
> > Copyright 2004 Free Software Foundation, Inc.
> > GDB is free software, covered by the GNU General Public License, and you
> are
> > welcome to change it and/or distribute copies of it under certain
> > conditions.
> > Type "show copying" to see the conditions.
> > There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> > This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols
> > found)...
> > Core was generated by `suricata'.
> > Program terminated with signal 11, Segmentation fault.
> > Reading symbols from /usr/local/lib/libprelude.so.2...(no debugging
> symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libprelude.so.2
> > Reading symbols from /usr/local/lib/libgnutls.so.28...(no debugging
> symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libgnutls.so.28
> > Reading symbols from /usr/local/lib/libgcrypt.so.20...(no debugging
> symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libgcrypt.so.20
> > Reading symbols from /usr/local/lib/libgpg-error.so.0...(no debugging
> > symbols found)...done.
> > Loaded symbols for /usr/local/lib/libgpg-error.so.0
> > Reading symbols from /usr/lib/libmagic.so.4...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/lib/libmagic.so.4
> > Reading symbols from /usr/local/lib/libhtp-0.5.16.so.1...(no debugging
> > symbols found)...done.
> > Loaded symbols for /usr/local/lib/libhtp-0.5.16.so.1
> > Reading symbols from /lib/libpcap.so.8...(no debugging symbols
> > found)...done.
> > Loaded symbols for /lib/libpcap.so.8
> > Reading symbols from /usr/local/lib/libnet11/libnet.so.1...(no debugging
> > symbols found)...done.
> > Loaded symbols for /usr/local/lib/libnet11/libnet.so.1
> > Reading symbols from /usr/local/lib/libjansson.so.4...(no debugging
> symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libjansson.so.4
> > Reading symbols from /lib/libthr.so.3...(no debugging symbols
> found)...done.
> > Loaded symbols for /lib/libthr.so.3
> > Reading symbols from /usr/local/lib/libyaml-0.so.2...(no debugging
> symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libyaml-0.so.2
> > Reading symbols from /usr/local/lib/libpcre.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libpcre.so.1
> > Reading symbols from /usr/local/lib/libplds4.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libplds4.so.1
> > Reading symbols from /usr/local/lib/libplc4.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libplc4.so.1
> > Reading symbols from /usr/local/lib/libnspr4.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libnspr4.so.1
> > Reading symbols from /lib/libc.so.7...(no debugging symbols
> found)...done.
> > Loaded symbols for /lib/libc.so.7
> > Reading symbols from /usr/local/lib/libltdl.so.7...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libltdl.so.7
> > Reading symbols from /lib/libz.so.6...(no debugging symbols
> found)...done.
> > Loaded symbols for /lib/libz.so.6
> > Reading symbols from /usr/local/lib/libp11-kit.so.0...(no debugging
> symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libp11-kit.so.0
> > Reading symbols from /usr/local/lib/libtspi.so.1...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libtspi.so.1
> > Reading symbols from /usr/local/lib/libtasn1.so.6...(no debugging symbols
> > found)...done.
> > Loaded symbols for /usr/local/lib/libtasn1.so.6
> > Reading symbols from /usr/local/lib/libnettle.so.4...done.
> > Loaded symbols for /usr/local/lib/libnettle.so.4
> > Reading symbols from /usr/local/lib/libhogweed.so.2...done.
> > Loaded symbols for /usr/local/lib/libhogweed.so.2
> > Reading symbols from /usr/local/lib/libgmp.so.10...done.
> > Loaded symbols for /usr/local/lib/libgmp.so.10
> > Reading symbols from /usr/local/lib/libintl.so.8...done.
> > Loaded symbols for /usr/local/lib/libintl.so.8
> > Reading symbols from /usr/local/lib/libiconv.so.2...done.
> > Loaded symbols for /usr/local/lib/libiconv.so.2
> > Reading symbols from /usr/local/lib/libffi.so.6...done.
> > Loaded symbols for /usr/local/lib/libffi.so.6
> > Reading symbols from /lib/libcrypto.so.7...done.
> > Loaded symbols for /lib/libcrypto.so.7
> > Reading symbols from /libexec/ld-elf.so.1...done.
> > Loaded symbols for /libexec/ld-elf.so.1
> > #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> > [New Thread 8107f4000 (LWP 101545/SCPerfMgmtThrea)]
> > [New Thread 8107f3c00 (LWP 100251/SCPerfWakeupThr)]
> > [New Thread 8107f3800 (LWP 100250/FlowManagerThre)]
> > [New Thread 8107f3400 (LWP 100249/Detect12)]
> > [New Thread 8107f3000 (LWP 100248/Detect11)]
> > [New Thread 8107f2c00 (LWP 100247/Detect10)]
> > [New Thread 8107f2800 (LWP 100246/Detect9)]
> > [New Thread 8107f2400 (LWP 100245/Detect8)]
> > [New Thread 8107f2000 (LWP 100244/Detect7)]
> > [New Thread 8107f1c00 (LWP 100243/Detect6)]
> > [New Thread 8107f1800 (LWP 100242/Detect5)]
> > [New Thread 8107f1400 (LWP 100240/Detect4)]
> > [New Thread 8107f1000 (LWP 100238/Detect3)]
> > [New Thread 8107f0c00 (LWP 100237/Detect2)]
> > [New Thread 8107f0800 (LWP 100236/Detect1)]
> > [New Thread 8107f0400 (LWP 100234/RxPcapbridge01)]
> > [New Thread 805415c00 (LWP 100229/RxPcapbridge21)]
> > [New Thread 805415800 (LWP 100164/RxPcapbridge11)]
> > [New Thread 805406400 (LWP 100600/suricata)]
> > (gdb) bt
> > #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> > #1  0x000000000045400f in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> > (gdb) frame 5
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> >
> > cat gdb.txt
> >
> > Thread 19 (Thread 805406400 (LWP 100600/suricata)):
> > #0  0x0000000802e930ea in _nanosleep () from /lib/libc.so.7
> > #1  0x000000080206cb0c in pthread_suspend_all_np () from /lib/libthr.so.3
> > #2  0x0000000802edb5f7 in usleep () from /lib/libc.so.7
> > #3  0x00000000005160e9 in ?? ()
> > #4  0x0000000000407e3f in ?? ()
> > #5  0x00000008007f1000 in ?? ()
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 18 (Thread 805415800 (LWP 100164/RxPcapbridge11)):
> > #0  0x0000000802efdd98 in _read () from /lib/libc.so.7
> > #1  0x000000080206cd46 in pthread_suspend_all_np () from /lib/libthr.so.3
> > #2  0x0000000801a24e0e in pcap_platform_finddevs () from
> /lib/libpcap.so.8
> > #3  0x00000000004ff26e in ?? ()
> > #4  0x000000000051f4bc in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 17 (Thread 805415c00 (LWP 100229/RxPcapbridge21)):
> > #0  0x0000000802075c5a in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x0000000802070b64 in pthread_mutex_destroy () from /lib/libthr.so.3
> > #2  0x00000000004c4ca4 in ?? ()
> > #3  0x00000000004c3031 in ?? ()
> > #4  0x0000000000440b80 in ?? ()
> > #5  0x000000000043ec1a in ?? ()
> > #6  0x000000000043d732 in ?? ()
> > #7  0x00000000004ffe3d in ?? ()
> > #8  0x000000000051f1cb in ?? ()
> > #9  0x00000000005000b1 in ?? ()
> > #10 0x0000000801a25394 in pcap_platform_finddevs () from
> /lib/libpcap.so.8
> > #11 0x00000000004ff26e in ?? ()
> > #12 0x000000000051f4bc in ?? ()
> > #13 0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #14 0x0000000000000000 in ?? ()
> >
> > Thread 16 (Thread 8107f0400 (LWP 100234/RxPcapbridge01)):
> > #0  0x00000000004410bb in ?? ()
> > #1  0x000000000043d781 in ?? ()
> > #2  0x00000000004ffe3d in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x00000000005000b1 in ?? ()
> > #5  0x0000000801a25394 in pcap_platform_finddevs () from
> /lib/libpcap.so.8
> > #6  0x00000000004ff26e in ?? ()
> > #7  0x000000000051f4bc in ?? ()
> > #8  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #9  0x0000000000000000 in ?? ()
> >
> > Thread 15 (Thread 8107f0800 (LWP 100236/Detect1)):
> > #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> > #1  0x000000000045400f in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 14 (Thread 8107f0c00 (LWP 100237/Detect2)):
> > #0  0x000000000054a90a in ?? ()
> > #1  0x0000000000454c00 in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 13 (Thread 8107f1000 (LWP 100238/Detect3)):
> > #0  0x000000000054a901 in ?? ()
> > #1  0x0000000000454c18 in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 12 (Thread 8107f1400 (LWP 100240/Detect4)):
> > #0  0x000000000054a901 in ?? ()
> > #1  0x0000000000454c18 in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 11 (Thread 8107f1800 (LWP 100242/Detect5)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x000000000051b41a in ?? ()
> > #3  0x000000000051f8c4 in ?? ()
> > #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #5  0x0000000000000000 in ?? ()
> >
> > Thread 10 (Thread 8107f1c00 (LWP 100243/Detect6)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x000000000051b41a in ?? ()
> > #3  0x000000000051f8c4 in ?? ()
> > #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #5  0x0000000000000000 in ?? ()
> >
> > Thread 9 (Thread 8107f2000 (LWP 100244/Detect7)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x000000000051b41a in ?? ()
> > #3  0x000000000051f8c4 in ?? ()
> > #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #5  0x0000000000000000 in ?? ()
> >
> > Thread 8 (Thread 8107f2400 (LWP 100245/Detect8)):
> > #0  0x0000000802070a48 in pthread_mutex_destroy () from /lib/libthr.so.3
> > #1  0x000000000045400f in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 7 (Thread 8107f2800 (LWP 100246/Detect9)):
> > #0  0x000000000054a901 in ?? ()
> > #1  0x0000000000454c00 in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 6 (Thread 8107f2c00 (LWP 100247/Detect10)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x000000000051b41a in ?? ()
> > #3  0x000000000051f8c4 in ?? ()
> > #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #5  0x0000000000000000 in ?? ()
> >
> > Thread 5 (Thread 8107f3000 (LWP 100248/Detect11)):
> > #0  0x000000000054a8d4 in ?? ()
> > #1  0x0000000000454c18 in ?? ()
> > #2  0x0000000000451d26 in ?? ()
> > #3  0x000000000051f1cb in ?? ()
> > #4  0x000000000051f8db in ?? ()
> > #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #6  0x0000000000000000 in ?? ()
> >
> > Thread 4 (Thread 8107f3400 (LWP 100249/Detect12)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x000000000051b41a in ?? ()
> > #3  0x000000000051f8c4 in ?? ()
> > #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #5  0x0000000000000000 in ?? ()
> >
> > Thread 3 (Thread 8107f3800 (LWP 100250/FlowManagerThre)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x00000000004c5adb in ?? ()
> > #3  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #4  0x0000000000000000 in ?? ()
> >
> > Thread 2 (Thread 8107f3c00 (LWP 100251/SCPerfWakeupThr)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x00000000004397e2 in ?? ()
> > #3  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #4  0x0000000000000000 in ?? ()
> >
> > Thread 1 (Thread 8107f4000 (LWP 101545/SCPerfMgmtThrea)):
> > #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> > #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> > #2  0x0000000000439ae7 in ?? ()
> > #3  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> > #4  0x0000000000000000 in ?? ()
> > #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> >
>
> You should recompile Suricata with debugging enabled (as explained in
> the link previously provided) - then after you get a core dump - do
> the output again:
>
> ./configure whatever_your_usual_flags_are CFLAGS="-ggdb -O0"
>
> So just add CFLAGS="-ggdb -O0" at the end of the configure line. After
> that the core - will be much more helpful.
>
> Thank you
>
>
>
OK just a follow-up, so far, Suricata did not die after I compiled with the
suggested flags.

Before recompiling, it was dying at least twice a day. Probably this much
improved stability is caused by -O0 or is just a coincidence and I will see
it dying again in the next days. However doesnt look like a coincidence to
me. If -O0 is causing more stability than -O2 I will try compiling it with
gcc (FreeBSD' s default is clang).

I'll share more updates as this evolves.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20150523/8e386bef/attachment-0002.html>