[Oisf-users] segfault in libc

[Victor Julien]

On 09/03/2013 03:33 PM, Christophe Vandeplas wrote:
> Hello,
> 
> The last few days I've got many segfaults with Suricata 1.4.5.
> It runs in afpacket mode. Unfortunately only 8 GB of RAM for 8 cores,
> waiting for more.
> 
> I unfortunately have no pcaps to reproduce the crash. However the
> crash happens daily now. I installed the -dbg packet and ran suri
> again to get a good core dump.
> 
> dmesg says:
> [Mon Sep  2 18:00:48 2013] Detect1[26101] general protection
> ip:7f0302c1af47 sp:7f02bfffd200 error:0 in
> libc-2.15.so[7f0302bea000+1b5000]
> [Mon Sep  2 18:02:04 2013] init: suricata main process (21845) killed
> by SEGV signal
> 
> When I load the core file:
> Program terminated with signal 11, Segmentation fault.
> #0  0x00007f0302c1af47 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> 
> (gdb) bt
> #0  0x00007f0302c1af47 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007f0302c1917e in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
> #2  0x00000000005ac84d in StreamTcpReassembleAppLayer (tv=0x23258570,
> ra_ctx=0x7f02b8001540, ssn=0x7f027c72c9e0,
>     stream=0x7f027c72ca30, p=0x25057a0) at stream-tcp-reassemble.c:2979
> #3  0x0000000000000000 in ?? ()
> 
> (gdb) frame 2
> #2  0x00000000005ac84d in StreamTcpReassembleAppLayer (tv=0x23258570,
> ra_ctx=0x7f02b8001540, ssn=0x7f027c72c9e0,
>     stream=0x7f027c72ca30, p=0x25057a0) at stream-tcp-reassemble.c:2979
> 2979 stream-tcp-reassemble.c: No such file or directory.
> 
> (gdb) info locals
> flags = 4 '\004'
> seg_tail = 0x7f02b22dd170
> ra_base_seq = 2651792181
> data = (upon request)
> data_len = 64360
> payload_offset = 0
> payload_len = 284
> next_seq = 2651726646
> seg = 0x0
> __PRETTY_FUNCTION__ = "StreamTcpReassembleAppLayer"
> 
> Any pointer what I should do next to help locate the problem?

This is really strange. The error you get is SEGV, but the bt says it
crashes on a line:
        BUG_ON(data_len > sizeof(data));

BUG_ON is a wrapper around assert, which should exit/crash with a
different signal. Weird.

To get at this point there has to be some bug during reassembly. The
data_len variable is too big, that is clear.

There are a few more 'BUG_ON' statements in that same function. They are
enabled by debug mode. Running with debugging enabled is probably not
feasible, so maybe you can edit the code to remove the various:

                if (SCLogDebugEnabled()) {

and their corresponding

		}

and then recompile. If it crashes slightly sooner, it may give us the
clue we need.

Alternatively, if you can just let me ssh into that box I might be able
to figure things out sooner, but thats unusually not an option :)

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------