[Oisf-users] segfault - smtp parser - suricata 1.1
Victor Julien
victor at inliniac.net
Sat Dec 3 08:32:56 UTC 2011
Hi Christophe, thanks for your report.
On 12/03/2011 08:47 AM, Christophe Vandeplas wrote:
> After running for 6 hours it segfaulted again without the rules I
> thought could cause this problem. (I had rules that contained Russian
> UTF chars)
>
> Here's a new backtrace, again at the same location in the code.
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0xb73c1b70 (LWP 25097)]
> 0xb7eb67aa in ?? () from /lib/i386-linux-gnu/libc.so.6
> (gdb) bt
> #0 0xb7eb67aa in ?? () from /lib/i386-linux-gnu/libc.so.6
> #1 0x081713bb in SMTPGetLine (state=0xde27870) at
> /usr/include/bits/string3.h:52
Can you do a "print *state" here? (do "f 1" first).
> #2 0x081763e8 in SMTPParse (f=0xad8a49e0, alstate=0xde27870, pstate=0xde27850,
> input=0xb73c00dc
> "\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
> input_len=1460, output=0xb73bf734) at app-layer-smtp.c:559
Can you do a "print *state" here as well? (do "f 2" first).
Also, can you do:
set print elements 0
set print repeats 0
print input
And send that to me? That should give me enough info to try and
reproduce the issue.
If you prefer, you can send this info privately of course!
Thanks!
Cheers,
Victor
> #3 SMTPParseClientRecord (f=0xad8a49e0, alstate=0xde27870, pstate=0xde27850,
> input=0xb73c00dc
> "\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
> input_len=1460, output=0xb73bf734) at app-layer-smtp.c:581
> #4 0x0815c66e in AppLayerDoParse (f=0xad8a49e0,
> app_layer_state=0xde27870, parser_state=0xde27850,
> input=0xb73c00dc
> "\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
> input_len=1460, parser_idx=16, proto=3) at app-layer-parser.c:695
> #5 0x0815f579 in AppLayerParse (f=0xad8a49e0, proto=3 '\003', flags=4 '\004',
> input=0xb73c00dc
> "\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
> input_len=1460) at app-layer-parser.c:908
> #6 0x0814bbf3 in StreamTcpReassembleAppLayer (ra_ctx=0xb6201eb0,
> ssn=0xaf6fc378, stream=0xaf6fc3b8, p=<value optimized out>) at
> stream-tcp-reassemble.c:3051
> #7 0x0814c551 in StreamTcpReassembleHandleSegmentUpdateACK
> (ra_ctx=0xb6201eb0, ssn=0xaf6fc378, stream=0xaf6fc3b8, p=0x8affd40) at
> stream-tcp-reassemble.c:3430
> #8 0x0814e0d5 in StreamTcpReassembleHandleSegment (tv=0xb338650,
> ra_ctx=0xb6201eb0, ssn=0xaf6fc378, stream=0xaf6fc37c, p=0x8affd40,
> pq=0xb6201948) at stream-tcp-reassemble.c:3504
> #9 0x08147c36 in HandleEstablishedPacketToClient (tv=0xb338650,
> p=<value optimized out>, stt=0xb6201940, pq=0xb2366e8) at
> stream-tcp.c:1826
> #10 StreamTcpPacketStateEstablished (tv=0xb338650, p=<value optimized
> out>, stt=0xb6201940, pq=0xb2366e8) at stream-tcp.c:1980
> #11 StreamTcpPacket (tv=0xb338650, p=<value optimized out>,
> stt=0xb6201940, pq=0xb2366e8) at stream-tcp.c:3560
> #12 0x08149691 in StreamTcp (tv=0xb338650, p=0x8affd40,
> data=0xb6201940, pq=0xb2366e8, postpq=0x0) at stream-tcp.c:3765
> #13 0x0812d444 in TmThreadsSlotVarRun (tv=0xb338650, p=0x8affd40,
> slot=0xb2bf160) at tm-threads.c:458
> #14 0x0812f8c7 in TmThreadsSlotVar (td=0xb338650) at tm-threads.c:655
> #15 0xb7f64e99 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
> #16 0xb7e7173e in clone () from /lib/i386-linux-gnu/libc.so.6
> (gdb) i r
> eax 0xb73c1ff7 -1220796425
> ecx 0xffffe064 -8092
> edx 0x8c410b0 147067056
> ebx 0xb7efdff4 -1209016332
> esp 0xb73bee48 0xb73bee48
> ebp 0xb73bf6a8 0xb73bf6a8
> esi 0xb73c00dc -1220804388
> edi 0x4c 76
> eip 0xb7eb67aa 0xb7eb67aa
> eflags 0x10282 [ SF IF RF ]
> cs 0x73 115
> ss 0x7b 123
> ds 0x7b 123
> es 0x7b 123
> fs 0x0 0
> gs 0x33 51
>
>
> On Fri, Dec 2, 2011 at 10:10 PM, Christophe Vandeplas
> <christophe at vandeplas.com> wrote:
>> Hello,
>>
>> I installed/compiled suricata v1.1 yesterday and had 2 segfaults over
>> a very short time.
>>
>> The segfaults I had were: (from dmesg)
>> [143691.171972] Decode & Stream[26656]: segfault at b6b17000 ip
>> b760b7aa sp b6b13e48 error 4 in libc-2.13.so[b74f6000+15a000]
>> [177583.326790] Decode & Stream[517]: segfault at b6c2a002 ip b771e7aa
>> sp b6c26e48 error 4 in libc-2.13.so[b7609000+15a000]
>>
>> So the third time I ran suricata I did it in a debugger and it crashed
>> again a few hours later.
>> Here is the backtrace:
>> Program received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0xb73c1b70 (LWP 27915)]
>> 0xb7eb67ae in ?? () from /lib/i386-linux-gnu/libc.so.6
>> (gdb)
>> (gdb) backtrace
>> #0 0xb7eb67ae in ?? () from /lib/i386-linux-gnu/libc.so.6
>> #1 0x081713bb in SMTPGetLine (state=0x9ea81a40) at
>> /usr/include/bits/string3.h:52
>> #2 0x081763e8 in SMTPParse (f=0xa98544d0, alstate=0x9ea81a40,
>> pstate=0xa5c5df88,
>> input=0xb73c00dc
>> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
>> input_len=2920, output=0xb73bf734) at app-layer-smtp.c:559
>> #3 SMTPParseClientRecord (f=0xa98544d0, alstate=0x9ea81a40, pstate=0xa5c5df88,
>> input=0xb73c00dc
>> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
>> input_len=2920, output=0xb73bf734) at app-layer-smtp.c:581
>> #4 0x0815c66e in AppLayerDoParse (f=0xa98544d0,
>> app_layer_state=0x9ea81a40, parser_state=0xa5c5df88,
>> input=0xb73c00dc
>> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
>> input_len=2920, parser_idx=16, proto=3) at app-layer-parser.c:695
>> #5 0x0815f579 in AppLayerParse (f=0xa98544d0, proto=3 '\003', flags=4 '\004',
>> input=0xb73c00dc
>> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
>> input_len=2920) at app-layer-parser.c:908
>> #6 0x0814bbf3 in StreamTcpReassembleAppLayer (ra_ctx=0xb6201f08,
>> ssn=0xaf458638, stream=0xaf458678, p=<value optimized out>) at
>> stream-tcp-reassemble.c:3051
>> #7 0x0814c551 in StreamTcpReassembleHandleSegmentUpdateACK
>> (ra_ctx=0xb6201f08, ssn=0xaf458638, stream=0xaf458678, p=0x8b02e58) at
>> stream-tcp-reassemble.c:3430
>> #8 0x0814e0d5 in StreamTcpReassembleHandleSegment (tv=0xb399888,
>> ra_ctx=0xb6201f08, ssn=0xaf458638, stream=0xaf45863c, p=0x8b02e58,
>> pq=0xb62019a0) at stream-tcp-reassemble.c:3504
>> #9 0x08147c36 in HandleEstablishedPacketToClient (tv=0xb399888,
>> p=<value optimized out>, stt=0xb6201998, pq=0xb2d8d50) at
>> stream-tcp.c:1826
>> #10 StreamTcpPacketStateEstablished (tv=0xb399888, p=<value optimized
>> out>, stt=0xb6201998, pq=0xb2d8d50) at stream-tcp.c:1980
>> #11 StreamTcpPacket (tv=0xb399888, p=<value optimized out>,
>> stt=0xb6201998, pq=0xb2d8d50) at stream-tcp.c:3560
>> #12 0x08149691 in StreamTcp (tv=0xb399888, p=0x8b02e58,
>> data=0xb6201998, pq=0xb2d8d50, postpq=0x0) at stream-tcp.c:3765
>> #13 0x0812d444 in TmThreadsSlotVarRun (tv=0xb399888, p=0x8b02e58,
>> slot=0xd14dda0) at tm-threads.c:458
>> #14 0x0812f8c7 in TmThreadsSlotVar (td=0xb399888) at tm-threads.c:655
>> #15 0xb7f64e99 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
>> #16 0xb7e7173e in clone () from /lib/i386-linux-gnu/libc.so.6
>> (gdb) i r
>> eax 0xb73c1fef -1220796433
>> ecx 0xffffe06c -8084
>> edx 0x98e3d5c0 -1729899072
>> ebx 0xb7efdff4 -1209016332
>> esp 0xb73bee48 0xb73bee48
>> ebp 0xb73bf6a8 0xb73bf6a8
>> esi 0xb73c00dc -1220804388
>> edi 0x4c 76
>> eip 0xb7eb67ae 0xb7eb67ae
>> eflags 0x10286 [ PF SF IF RF ]
>> cs 0x73 115
>> ss 0x7b 123
>> ds 0x7b 123
>> es 0x7b 123
>> fs 0x0 0
>> gs 0x33 51
>>
>>
>> It looks like the SMTP parser is crashing, but as I'm not really that
>> good with gdb I don't know what else I should do to pinpoint the
>> cause.
>> Unfortunately I can't take network captures as I'm working with data
>> from a production environment.(privacy and confidentiality) (and it
>> takes a few hours to crash)
>> Bandwidth was around 10-20 Mbit/s, sniffing a mirror port with
>> multiple (tagged) VLANs
>>
>> Could you tell me what I should do (in gdb) to help us understand the
>> origin of the problem ?
>>
>> In the meantime I'll keep gdb open with the crashed instance of suricata.
>> And will try disabling some rules to check if that might be the origin
>> of the segfault.
>>
>> Thanks
>> Christophe
> _______________________________________________
> Oisf-users mailing list
> Oisf-users at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
More information about the Oisf-users
mailing list