[Oisf-users] segfault - smtp parser - suricata 1.1
Christophe Vandeplas
christophe at vandeplas.com
Sat Dec 3 07:47:01 UTC 2011
After running for 6 hours it segfaulted again without the rules I
thought could cause this problem. (I had rules that contained Russian
UTF chars)
Here's a new backtrace, again at the same location in the code.
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xb73c1b70 (LWP 25097)]
0xb7eb67aa in ?? () from /lib/i386-linux-gnu/libc.so.6
(gdb) bt
#0 0xb7eb67aa in ?? () from /lib/i386-linux-gnu/libc.so.6
#1 0x081713bb in SMTPGetLine (state=0xde27870) at
/usr/include/bits/string3.h:52
#2 0x081763e8 in SMTPParse (f=0xad8a49e0, alstate=0xde27870, pstate=0xde27850,
input=0xb73c00dc
"\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
input_len=1460, output=0xb73bf734) at app-layer-smtp.c:559
#3 SMTPParseClientRecord (f=0xad8a49e0, alstate=0xde27870, pstate=0xde27850,
input=0xb73c00dc
"\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
input_len=1460, output=0xb73bf734) at app-layer-smtp.c:581
#4 0x0815c66e in AppLayerDoParse (f=0xad8a49e0,
app_layer_state=0xde27870, parser_state=0xde27850,
input=0xb73c00dc
"\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
input_len=1460, parser_idx=16, proto=3) at app-layer-parser.c:695
#5 0x0815f579 in AppLayerParse (f=0xad8a49e0, proto=3 '\003', flags=4 '\004',
input=0xb73c00dc
"\nYWdlcy9EZVRpamQvRGVUaWpkTW9ybmluZ0NvbW1lbnQvYXJyb3cuanBnIj48L2I+PC9mb250Pjwv\r\ndGQ+DQogPC90cj4NCiA8L3RhYmxlPg0KIA0KIDx0YWJsZSBjZWxscGFkZGluZz0iMCIgY2Vsc3Bh\r\nY2luZz0iMCIgYm9yZGVyPSIwIiB3aWR0aD0iMTAwJSI"...,
input_len=1460) at app-layer-parser.c:908
#6 0x0814bbf3 in StreamTcpReassembleAppLayer (ra_ctx=0xb6201eb0,
ssn=0xaf6fc378, stream=0xaf6fc3b8, p=<value optimized out>) at
stream-tcp-reassemble.c:3051
#7 0x0814c551 in StreamTcpReassembleHandleSegmentUpdateACK
(ra_ctx=0xb6201eb0, ssn=0xaf6fc378, stream=0xaf6fc3b8, p=0x8affd40) at
stream-tcp-reassemble.c:3430
#8 0x0814e0d5 in StreamTcpReassembleHandleSegment (tv=0xb338650,
ra_ctx=0xb6201eb0, ssn=0xaf6fc378, stream=0xaf6fc37c, p=0x8affd40,
pq=0xb6201948) at stream-tcp-reassemble.c:3504
#9 0x08147c36 in HandleEstablishedPacketToClient (tv=0xb338650,
p=<value optimized out>, stt=0xb6201940, pq=0xb2366e8) at
stream-tcp.c:1826
#10 StreamTcpPacketStateEstablished (tv=0xb338650, p=<value optimized
out>, stt=0xb6201940, pq=0xb2366e8) at stream-tcp.c:1980
#11 StreamTcpPacket (tv=0xb338650, p=<value optimized out>,
stt=0xb6201940, pq=0xb2366e8) at stream-tcp.c:3560
#12 0x08149691 in StreamTcp (tv=0xb338650, p=0x8affd40,
data=0xb6201940, pq=0xb2366e8, postpq=0x0) at stream-tcp.c:3765
#13 0x0812d444 in TmThreadsSlotVarRun (tv=0xb338650, p=0x8affd40,
slot=0xb2bf160) at tm-threads.c:458
#14 0x0812f8c7 in TmThreadsSlotVar (td=0xb338650) at tm-threads.c:655
#15 0xb7f64e99 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
#16 0xb7e7173e in clone () from /lib/i386-linux-gnu/libc.so.6
(gdb) i r
eax 0xb73c1ff7 -1220796425
ecx 0xffffe064 -8092
edx 0x8c410b0 147067056
ebx 0xb7efdff4 -1209016332
esp 0xb73bee48 0xb73bee48
ebp 0xb73bf6a8 0xb73bf6a8
esi 0xb73c00dc -1220804388
edi 0x4c 76
eip 0xb7eb67aa 0xb7eb67aa
eflags 0x10282 [ SF IF RF ]
cs 0x73 115
ss 0x7b 123
ds 0x7b 123
es 0x7b 123
fs 0x0 0
gs 0x33 51
On Fri, Dec 2, 2011 at 10:10 PM, Christophe Vandeplas
<christophe at vandeplas.com> wrote:
> Hello,
>
> I installed/compiled suricata v1.1 yesterday and had 2 segfaults over
> a very short time.
>
> The segfaults I had were: (from dmesg)
> [143691.171972] Decode & Stream[26656]: segfault at b6b17000 ip
> b760b7aa sp b6b13e48 error 4 in libc-2.13.so[b74f6000+15a000]
> [177583.326790] Decode & Stream[517]: segfault at b6c2a002 ip b771e7aa
> sp b6c26e48 error 4 in libc-2.13.so[b7609000+15a000]
>
> So the third time I ran suricata I did it in a debugger and it crashed
> again a few hours later.
> Here is the backtrace:
> Program received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0xb73c1b70 (LWP 27915)]
> 0xb7eb67ae in ?? () from /lib/i386-linux-gnu/libc.so.6
> (gdb)
> (gdb) backtrace
> #0 0xb7eb67ae in ?? () from /lib/i386-linux-gnu/libc.so.6
> #1 0x081713bb in SMTPGetLine (state=0x9ea81a40) at
> /usr/include/bits/string3.h:52
> #2 0x081763e8 in SMTPParse (f=0xa98544d0, alstate=0x9ea81a40,
> pstate=0xa5c5df88,
> input=0xb73c00dc
> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
> input_len=2920, output=0xb73bf734) at app-layer-smtp.c:559
> #3 SMTPParseClientRecord (f=0xa98544d0, alstate=0x9ea81a40, pstate=0xa5c5df88,
> input=0xb73c00dc
> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
> input_len=2920, output=0xb73bf734) at app-layer-smtp.c:581
> #4 0x0815c66e in AppLayerDoParse (f=0xa98544d0,
> app_layer_state=0x9ea81a40, parser_state=0xa5c5df88,
> input=0xb73c00dc
> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
> input_len=2920, parser_idx=16, proto=3) at app-layer-parser.c:695
> #5 0x0815f579 in AppLayerParse (f=0xa98544d0, proto=3 '\003', flags=4 '\004',
> input=0xb73c00dc
> "\nxsPHQce/yD3IvMk6ybnKOMq3yzbLtsw1zLXNNc21zjbOts83z7jQOdC60TzRvtI/0sHTRNPG1EnU\r\ny9VO1dHWVdbY11zX4Nhk2OjZbNnx2nba+9uA3AXcit0Q3ZbeHN6i3ynfr+A24L3hROHM4lPi2+Nj\r\n4+vkc+T85YTmDeaW5x/nqegy6LzpRunQ6lvq5etw6/v"...,
> input_len=2920) at app-layer-parser.c:908
> #6 0x0814bbf3 in StreamTcpReassembleAppLayer (ra_ctx=0xb6201f08,
> ssn=0xaf458638, stream=0xaf458678, p=<value optimized out>) at
> stream-tcp-reassemble.c:3051
> #7 0x0814c551 in StreamTcpReassembleHandleSegmentUpdateACK
> (ra_ctx=0xb6201f08, ssn=0xaf458638, stream=0xaf458678, p=0x8b02e58) at
> stream-tcp-reassemble.c:3430
> #8 0x0814e0d5 in StreamTcpReassembleHandleSegment (tv=0xb399888,
> ra_ctx=0xb6201f08, ssn=0xaf458638, stream=0xaf45863c, p=0x8b02e58,
> pq=0xb62019a0) at stream-tcp-reassemble.c:3504
> #9 0x08147c36 in HandleEstablishedPacketToClient (tv=0xb399888,
> p=<value optimized out>, stt=0xb6201998, pq=0xb2d8d50) at
> stream-tcp.c:1826
> #10 StreamTcpPacketStateEstablished (tv=0xb399888, p=<value optimized
> out>, stt=0xb6201998, pq=0xb2d8d50) at stream-tcp.c:1980
> #11 StreamTcpPacket (tv=0xb399888, p=<value optimized out>,
> stt=0xb6201998, pq=0xb2d8d50) at stream-tcp.c:3560
> #12 0x08149691 in StreamTcp (tv=0xb399888, p=0x8b02e58,
> data=0xb6201998, pq=0xb2d8d50, postpq=0x0) at stream-tcp.c:3765
> #13 0x0812d444 in TmThreadsSlotVarRun (tv=0xb399888, p=0x8b02e58,
> slot=0xd14dda0) at tm-threads.c:458
> #14 0x0812f8c7 in TmThreadsSlotVar (td=0xb399888) at tm-threads.c:655
> #15 0xb7f64e99 in start_thread () from /lib/i386-linux-gnu/libpthread.so.0
> #16 0xb7e7173e in clone () from /lib/i386-linux-gnu/libc.so.6
> (gdb) i r
> eax 0xb73c1fef -1220796433
> ecx 0xffffe06c -8084
> edx 0x98e3d5c0 -1729899072
> ebx 0xb7efdff4 -1209016332
> esp 0xb73bee48 0xb73bee48
> ebp 0xb73bf6a8 0xb73bf6a8
> esi 0xb73c00dc -1220804388
> edi 0x4c 76
> eip 0xb7eb67ae 0xb7eb67ae
> eflags 0x10286 [ PF SF IF RF ]
> cs 0x73 115
> ss 0x7b 123
> ds 0x7b 123
> es 0x7b 123
> fs 0x0 0
> gs 0x33 51
>
>
> It looks like the SMTP parser is crashing, but as I'm not really that
> good with gdb I don't know what else I should do to pinpoint the
> cause.
> Unfortunately I can't take network captures as I'm working with data
> from a production environment.(privacy and confidentiality) (and it
> takes a few hours to crash)
> Bandwidth was around 10-20 Mbit/s, sniffing a mirror port with
> multiple (tagged) VLANs
>
> Could you tell me what I should do (in gdb) to help us understand the
> origin of the problem ?
>
> In the meantime I'll keep gdb open with the crashed instance of suricata.
> And will try disabling some rules to check if that might be the origin
> of the segfault.
>
> Thanks
> Christophe
More information about the Oisf-users
mailing list