[Oisf-users] segfault in libc

[Anoop Saldanha]

On Tue, Sep 3, 2013 at 9:11 PM, Victor Julien <lists at inliniac.net> wrote:
> On 09/03/2013 03:33 PM, Christophe Vandeplas wrote:
>> Hello,
>>
>> The last few days I've got many segfaults with Suricata 1.4.5.
>> It runs in afpacket mode. Unfortunately only 8 GB of RAM for 8 cores,
>> waiting for more.
>>
>> I unfortunately have no pcaps to reproduce the crash. However the
>> crash happens daily now. I installed the -dbg packet and ran suri
>> again to get a good core dump.
>>
>> dmesg says:
>> [Mon Sep  2 18:00:48 2013] Detect1[26101] general protection
>> ip:7f0302c1af47 sp:7f02bfffd200 error:0 in
>> libc-2.15.so[7f0302bea000+1b5000]
>> [Mon Sep  2 18:02:04 2013] init: suricata main process (21845) killed
>> by SEGV signal
>>
>> When I load the core file:
>> Program terminated with signal 11, Segmentation fault.
>> #0  0x00007f0302c1af47 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>>
>> (gdb) bt
>> #0  0x00007f0302c1af47 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
>> #1  0x00007f0302c1917e in __assert_fail () from /lib/x86_64-linux-gnu/libc.so.6
>> #2  0x00000000005ac84d in StreamTcpReassembleAppLayer (tv=0x23258570,
>> ra_ctx=0x7f02b8001540, ssn=0x7f027c72c9e0,
>>     stream=0x7f027c72ca30, p=0x25057a0) at stream-tcp-reassemble.c:2979
>> #3  0x0000000000000000 in ?? ()
>>
>> (gdb) frame 2
>> #2  0x00000000005ac84d in StreamTcpReassembleAppLayer (tv=0x23258570,
>> ra_ctx=0x7f02b8001540, ssn=0x7f027c72c9e0,
>>     stream=0x7f027c72ca30, p=0x25057a0) at stream-tcp-reassemble.c:2979
>> 2979 stream-tcp-reassemble.c: No such file or directory.
>>
>> (gdb) info locals
>> flags = 4 '\004'
>> seg_tail = 0x7f02b22dd170
>> ra_base_seq = 2651792181
>> data = (upon request)
>> data_len = 64360
>> payload_offset = 0
>> payload_len = 284
>> next_seq = 2651726646
>> seg = 0x0
>> __PRETTY_FUNCTION__ = "StreamTcpReassembleAppLayer"
>>
>> Any pointer what I should do next to help locate the problem?
>
> This is really strange. The error you get is SEGV, but the bt says it
> crashes on a line:
>         BUG_ON(data_len > sizeof(data));
>
> BUG_ON is a wrapper around assert, which should exit/crash with a
> different signal. Weird.
>
> To get at this point there has to be some bug during reassembly. The
> data_len variable is too big, that is clear.
>
> There are a few more 'BUG_ON' statements in that same function. They are
> enabled by debug mode. Running with debugging enabled is probably not
> feasible, so maybe you can edit the code to remove the various:
>
>                 if (SCLogDebugEnabled()) {
>
> and their corresponding
>
>                 }
>
> and then recompile. If it crashes slightly sooner, it may give us the
> clue we need.
>
> Alternatively, if you can just let me ssh into that box I might be able
> to figure things out sooner, but thats unusually not an option :)
>

Maybe if we can get a capture it would make things easier as well.

-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------