[Oisf-devel] Suricata dies (core dump) w/ multiple NICs

Thu May 21 21:18:40 UTC 2015

On Thu, May 21, 2015 at 11:12 PM, Eduardo Meyer <dudu.meyer at gmail.com> wrote:
> On Thu, May 21, 2015 at 6:10 AM, Peter Manev <petermanev at gmail.com> wrote:
>>
>> On Thu, May 21, 2015 at 1:02 AM, Eduardo Meyer <dudu.meyer at gmail.com>
>> wrote:
>> > Hello,
>> >
>> > I am running Suricata 2.0.8 RELEASE with 3 interfaces, and from times to
>> > times suricata simply dies. This is the process arguments in use:
>> >
>> > root         45492   1.0  1.5 1299164 251564  -  Is    4:20PM
>> > 84:38.13
>> > /usr/local/bin/suricata -D -i bridge1 -i bridge2 -i bridge0 --pidfile
>> > /var/run/suricata_bridge0.pid -c /usr/local/etc/suricata/suricata.yaml
>> >
>> > I could not find a pattern when Suricata dies. Sometimes it's a high
>> > pps/memory/bandwidth usage profile, sometimes it's a low demand hour
>> > with
>> > just a couple pps passing the suricata system.
>> >
>> > It never dies with a single interface. It dies for bridged ports,
>> > trunked
>> > ports as well as for physical untagged ports, so it does not seem to be
>> > related to virtual or real NICs it's listening at, although I noticed it
>> > dies more frequently on bridged interfaces like the above scenario.
>> >
>> > Is there anything I should look at with special attention on
>> > suricata.yaml?
>> >
>> > I have a suricata.core everytime it dies. How can I produce useful
>> > information from it?
>>
>> If you have a core dump and can reproduce the issue consistently - you
>> can have a look at this guide here -
>>
>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Reporting_Bugs
>> how to extract useful info. Then you can open a bug report should you
>> consider.
>>
>> Thank you
>
>
>
> What else should I do to the bug the cause? I am no gdb familiarized in any
> ways, so I can't move forth, so far this is what I had only:
>
> gdb /usr/local/bin/suricata /suricata.core
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...(no debugging symbols
> found)...
> Core was generated by `suricata'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /usr/local/lib/libprelude.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libprelude.so.2
> Reading symbols from /usr/local/lib/libgnutls.so.28...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libgnutls.so.28
> Reading symbols from /usr/local/lib/libgcrypt.so.20...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libgcrypt.so.20
> Reading symbols from /usr/local/lib/libgpg-error.so.0...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/local/lib/libgpg-error.so.0
> Reading symbols from /usr/lib/libmagic.so.4...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/lib/libmagic.so.4
> Reading symbols from /usr/local/lib/libhtp-0.5.16.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/local/lib/libhtp-0.5.16.so.1
> Reading symbols from /lib/libpcap.so.8...(no debugging symbols
> found)...done.
> Loaded symbols for /lib/libpcap.so.8
> Reading symbols from /usr/local/lib/libnet11/libnet.so.1...(no debugging
> symbols found)...done.
> Loaded symbols for /usr/local/lib/libnet11/libnet.so.1
> Reading symbols from /usr/local/lib/libjansson.so.4...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libjansson.so.4
> Reading symbols from /lib/libthr.so.3...(no debugging symbols found)...done.
> Loaded symbols for /lib/libthr.so.3
> Reading symbols from /usr/local/lib/libyaml-0.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libyaml-0.so.2
> Reading symbols from /usr/local/lib/libpcre.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libpcre.so.1
> Reading symbols from /usr/local/lib/libplds4.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libplds4.so.1
> Reading symbols from /usr/local/lib/libplc4.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libplc4.so.1
> Reading symbols from /usr/local/lib/libnspr4.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libnspr4.so.1
> Reading symbols from /lib/libc.so.7...(no debugging symbols found)...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /usr/local/lib/libltdl.so.7...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libltdl.so.7
> Reading symbols from /lib/libz.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib/libz.so.6
> Reading symbols from /usr/local/lib/libp11-kit.so.0...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libp11-kit.so.0
> Reading symbols from /usr/local/lib/libtspi.so.1...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libtspi.so.1
> Reading symbols from /usr/local/lib/libtasn1.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /usr/local/lib/libtasn1.so.6
> Reading symbols from /usr/local/lib/libnettle.so.4...done.
> Loaded symbols for /usr/local/lib/libnettle.so.4
> Reading symbols from /usr/local/lib/libhogweed.so.2...done.
> Loaded symbols for /usr/local/lib/libhogweed.so.2
> Reading symbols from /usr/local/lib/libgmp.so.10...done.
> Loaded symbols for /usr/local/lib/libgmp.so.10
> Reading symbols from /usr/local/lib/libintl.so.8...done.
> Loaded symbols for /usr/local/lib/libintl.so.8
> Reading symbols from /usr/local/lib/libiconv.so.2...done.
> Loaded symbols for /usr/local/lib/libiconv.so.2
> Reading symbols from /usr/local/lib/libffi.so.6...done.
> Loaded symbols for /usr/local/lib/libffi.so.6
> Reading symbols from /lib/libcrypto.so.7...done.
> Loaded symbols for /lib/libcrypto.so.7
> Reading symbols from /libexec/ld-elf.so.1...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> [New Thread 8107f4000 (LWP 101545/SCPerfMgmtThrea)]
> [New Thread 8107f3c00 (LWP 100251/SCPerfWakeupThr)]
> [New Thread 8107f3800 (LWP 100250/FlowManagerThre)]
> [New Thread 8107f3400 (LWP 100249/Detect12)]
> [New Thread 8107f3000 (LWP 100248/Detect11)]
> [New Thread 8107f2c00 (LWP 100247/Detect10)]
> [New Thread 8107f2800 (LWP 100246/Detect9)]
> [New Thread 8107f2400 (LWP 100245/Detect8)]
> [New Thread 8107f2000 (LWP 100244/Detect7)]
> [New Thread 8107f1c00 (LWP 100243/Detect6)]
> [New Thread 8107f1800 (LWP 100242/Detect5)]
> [New Thread 8107f1400 (LWP 100240/Detect4)]
> [New Thread 8107f1000 (LWP 100238/Detect3)]
> [New Thread 8107f0c00 (LWP 100237/Detect2)]
> [New Thread 8107f0800 (LWP 100236/Detect1)]
> [New Thread 8107f0400 (LWP 100234/RxPcapbridge01)]
> [New Thread 805415c00 (LWP 100229/RxPcapbridge21)]
> [New Thread 805415800 (LWP 100164/RxPcapbridge11)]
> [New Thread 805406400 (LWP 100600/suricata)]
> (gdb) bt
> #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> #1  0x000000000045400f in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
> (gdb) frame 5
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
>
> cat gdb.txt
>
> Thread 19 (Thread 805406400 (LWP 100600/suricata)):
> #0  0x0000000802e930ea in _nanosleep () from /lib/libc.so.7
> #1  0x000000080206cb0c in pthread_suspend_all_np () from /lib/libthr.so.3
> #2  0x0000000802edb5f7 in usleep () from /lib/libc.so.7
> #3  0x00000000005160e9 in ?? ()
> #4  0x0000000000407e3f in ?? ()
> #5  0x00000008007f1000 in ?? ()
> #6  0x0000000000000000 in ?? ()
>
> Thread 18 (Thread 805415800 (LWP 100164/RxPcapbridge11)):
> #0  0x0000000802efdd98 in _read () from /lib/libc.so.7
> #1  0x000000080206cd46 in pthread_suspend_all_np () from /lib/libthr.so.3
> #2  0x0000000801a24e0e in pcap_platform_finddevs () from /lib/libpcap.so.8
> #3  0x00000000004ff26e in ?? ()
> #4  0x000000000051f4bc in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 17 (Thread 805415c00 (LWP 100229/RxPcapbridge21)):
> #0  0x0000000802075c5a in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x0000000802070b64 in pthread_mutex_destroy () from /lib/libthr.so.3
> #2  0x00000000004c4ca4 in ?? ()
> #3  0x00000000004c3031 in ?? ()
> #4  0x0000000000440b80 in ?? ()
> #5  0x000000000043ec1a in ?? ()
> #6  0x000000000043d732 in ?? ()
> #7  0x00000000004ffe3d in ?? ()
> #8  0x000000000051f1cb in ?? ()
> #9  0x00000000005000b1 in ?? ()
> #10 0x0000000801a25394 in pcap_platform_finddevs () from /lib/libpcap.so.8
> #11 0x00000000004ff26e in ?? ()
> #12 0x000000000051f4bc in ?? ()
> #13 0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #14 0x0000000000000000 in ?? ()
>
> Thread 16 (Thread 8107f0400 (LWP 100234/RxPcapbridge01)):
> #0  0x00000000004410bb in ?? ()
> #1  0x000000000043d781 in ?? ()
> #2  0x00000000004ffe3d in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x00000000005000b1 in ?? ()
> #5  0x0000000801a25394 in pcap_platform_finddevs () from /lib/libpcap.so.8
> #6  0x00000000004ff26e in ?? ()
> #7  0x000000000051f4bc in ?? ()
> #8  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #9  0x0000000000000000 in ?? ()
>
> Thread 15 (Thread 8107f0800 (LWP 100236/Detect1)):
> #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
> #1  0x000000000045400f in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 14 (Thread 8107f0c00 (LWP 100237/Detect2)):
> #0  0x000000000054a90a in ?? ()
> #1  0x0000000000454c00 in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 13 (Thread 8107f1000 (LWP 100238/Detect3)):
> #0  0x000000000054a901 in ?? ()
> #1  0x0000000000454c18 in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 12 (Thread 8107f1400 (LWP 100240/Detect4)):
> #0  0x000000000054a901 in ?? ()
> #1  0x0000000000454c18 in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 11 (Thread 8107f1800 (LWP 100242/Detect5)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x000000000051b41a in ?? ()
> #3  0x000000000051f8c4 in ?? ()
> #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #5  0x0000000000000000 in ?? ()
>
> Thread 10 (Thread 8107f1c00 (LWP 100243/Detect6)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x000000000051b41a in ?? ()
> #3  0x000000000051f8c4 in ?? ()
> #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #5  0x0000000000000000 in ?? ()
>
> Thread 9 (Thread 8107f2000 (LWP 100244/Detect7)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x000000000051b41a in ?? ()
> #3  0x000000000051f8c4 in ?? ()
> #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #5  0x0000000000000000 in ?? ()
>
> Thread 8 (Thread 8107f2400 (LWP 100245/Detect8)):
> #0  0x0000000802070a48 in pthread_mutex_destroy () from /lib/libthr.so.3
> #1  0x000000000045400f in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 7 (Thread 8107f2800 (LWP 100246/Detect9)):
> #0  0x000000000054a901 in ?? ()
> #1  0x0000000000454c00 in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 6 (Thread 8107f2c00 (LWP 100247/Detect10)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x000000000051b41a in ?? ()
> #3  0x000000000051f8c4 in ?? ()
> #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #5  0x0000000000000000 in ?? ()
>
> Thread 5 (Thread 8107f3000 (LWP 100248/Detect11)):
> #0  0x000000000054a8d4 in ?? ()
> #1  0x0000000000454c18 in ?? ()
> #2  0x0000000000451d26 in ?? ()
> #3  0x000000000051f1cb in ?? ()
> #4  0x000000000051f8db in ?? ()
> #5  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #6  0x0000000000000000 in ?? ()
>
> Thread 4 (Thread 8107f3400 (LWP 100249/Detect12)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x000000000051b41a in ?? ()
> #3  0x000000000051f8c4 in ?? ()
> #4  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #5  0x0000000000000000 in ?? ()
>
> Thread 3 (Thread 8107f3800 (LWP 100250/FlowManagerThre)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x00000000004c5adb in ?? ()
> #3  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #4  0x0000000000000000 in ?? ()
>
> Thread 2 (Thread 8107f3c00 (LWP 100251/SCPerfWakeupThr)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x00000000004397e2 in ?? ()
> #3  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #4  0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 8107f4000 (LWP 101545/SCPerfMgmtThrea)):
> #0  0x0000000802075c5c in pthread_cleanup_pop () from /lib/libthr.so.3
> #1  0x000000080207411e in _pthread_cond_wait () from /lib/libthr.so.3
> #2  0x0000000000439ae7 in ?? ()
> #3  0x000000080206a725 in pthread_create () from /lib/libthr.so.3
> #4  0x0000000000000000 in ?? ()
> #0  0x000000080207030a in pthread_mutex_lock () from /lib/libthr.so.3
>

You should recompile Suricata with debugging enabled (as explained in
the link previously provided) - then after you get a core dump - do
the output again:

./configure whatever_your_usual_flags_are CFLAGS="-ggdb -O0"

So just add CFLAGS="-ggdb -O0" at the end of the configure line. After
that the core - will be much more helpful.

Thank you

> --
> ===========
> Eduardo Meyer
> pessoal: dudu.meyer at gmail.com
> profissional: ddm.farmaciap at saude.gov.br

-- 
Regards,
Peter Manev