[Oisf-devel] Suricata 2.0dev + PF_RING 5.6.0 sporadic crashes in HTPCallbackRequest
Chris Wakelin
c.d.wakelin at reading.ac.uk
Mon Jul 22 12:55:11 UTC 2013
On 22/07/13 12:13, Victor Julien wrote:
> On 07/22/2013 11:13 AM, Chris Wakelin wrote:
>> On 19/07/13 13:58, Anoop Saldanha wrote:
>>> On Fri, Jul 19, 2013 at 6:07 PM, Chris Wakelin
>>> <c.d.wakelin at reading.ac.uk> wrote:
>>>> Hi,
>>>>
>>>> I recently upgraded our Suricata instances to Suricata 2.0dev (rev
>>>> 6229bfa - just a bit before the libhtp unbundling changes) and from
>>>> PF_RING 5.5.2 to 5.6.0.
>>>>
>>>> We're getting sporadic crashes in both sensors; they can go for a day
>>>> without crashing, then crash three times in half an hour, so it looks
>>>> like it's triggered by some very specific traffic.
>>>>
>>>> Looking in the backtrace, the ReceivePfringLoop frame suggests to me the
>>>> packet is corrupt (as far as I can tell - e.g. src and dst port are 0
>>>> and protocol 127, IPv4 addresses don't look like our local ones).
>>>>
>>>>> #17 0x0000000000592c0b in ReceivePfringLoop (tv=0xc971540, data=0x7fa92cd51f00, slot=0x86ca8c0) at source-pfring.c:311
>>>>> r = 1
>>>>> packet_q_len = 4989
>>>>> ptv = 0x7fa92cd51f00
>>>>> p = 0x420d400
>>>>> hdr = {ts = {tv_sec = 1374235900, tv_usec = 231639}, caplen = 60, len = 60, extended_hdr = {timestamp_ns = 4314826383835889664, rx_direction = 1 '\001', if_index = 6, pkt_hash = 1191440499, tx = {bounce_interface = 764450176, reserved = 0x3be157bc34058000},
>>>>> parsed_header_len = 0, parsed_pkt = {dmac = "\000\000\000\000\000", smac = "\000\000\000\000\000", eth_type = 38272, vlan_id = 11664, ip_version = 166 '�', l3_proto = 127 '\177', ip_tos = 0 '\000', ip_src = {v6 = {__in6_u = {
>>>>> __u6_addr8 = "�}F\000\000\000\000\000@\025\227\f\000\000\000", __u6_addr16 = {32243, 70, 0, 0, 5440, 3223, 0, 0}, __u6_addr32 = {4619763, 0, 211227968, 0}}}, v4 = 4619763}, ip_dst = {v6 = {__in6_u = {
>>>>> __u6_addr8 = "\000\200\005\064�W�;\000\000\000\000\000\000\000", __u6_addr16 = {32768, 13317, 22460, 15329, 0, 0, 0, 0}, __u6_addr32 = {872775680, 1004623804, 0, 0}}}, v4 = 872775680}, l4_src_port = 0, l4_dst_port = 0, tcp = {flags = 0 '\000',
>>>>> seq_num = 764450208, ack_num = 32678}, tunnel = {tunnel_id = 4626108, tunneled_proto = 0 '\000', tunneled_ip_src = {v6 = {__in6_u = {__u6_addr8 = "@\025\227\f\000\000\000\000Y()ު\177\000", __u6_addr16 = {5440, 3223, 0, 0, 10329, 56873, 32682, 0},
>>>>> __u6_addr32 = {211227968, 0, 3727239257, 32682}}}, v4 = 211227968}, tunneled_ip_dst = {v6 = {__in6_u = {__u6_addr8 = "\000\000\000\000\000\000\000\000\204\a,ު\177\000", __u6_addr16 = {0, 0, 0, 0, 1924, 56876, 32682, 0}, __u6_addr32 = {0, 0,
>>>>> 3727427460, 32682}}}, v4 = 0}, tunneled_l4_src_port = 0, tunneled_l4_dst_port = 0}, last_matched_plugin_id = 4, last_matched_rule_id = 0, offset = {eth_offset = 5440, vlan_offset = 3223, l3_offset = 0, l4_offset = 0, payload_offset = -27168}}}}
>>>>> s = 0x86ca8c0
>>>>> last_dump = 1374235900
>>>>> current_time = {tv_sec = 1374235900, tv_usec = 231263}
>>>>> __FUNCTION__ = "ReceivePfringLoop"
>>>>
>>>> I've attached a backtrace from a core that was generated a few minutes
>>>> ago (Suricata was compiled with CFLAGS="-ggdb -O0").
>>>>
>>>> Any ideas what traffic caused this? (My feeling is the corrupt packets,
>>>> if that's what they are, are probably PF_RING's fault, but of course
>>>> Suricata shouldn't crash even then.)
>>>>
>>>> I can downgrade Suricata, but alas I'm not allowed to touch PF_RING
>>>> without going through a Change Control process (it upset the border
>>>> switch once).
>>>>
>>>
>>> Can you run the lastest master(post 0.5.x changes). There were some
>>> bugs in libhtp which were fixed explicitly for 1.4.x, and for the
>>> master we relied on the 0.5.x fixing it.
>>>
>>
>> Still crashing sporadically I'm afraid, but now it's mostly in
>> htp_validate_hostname. I've attached another backtrace - does the frame
>> in ReceivePfringLoop make any sense?
>
> I agree the pfring data looks weird. Might be uninitialized, maybe
> pfring doesn't init it if it doesn't use it. Despite this value, suri
> figured out it's TCP.
>
I did as Ivan suggested and upgraded to latest libhtp git (I was running
master from Friday 19th July) and latest Suricata too (likewise)
Here's a backtrace for another crash we've been seeing a bit less
frequently but has just re-occurred (with the latest git releases).
Again the ReceivePfringLoop looks a bit suspect to me :)
Best Wishes,
Chris
--
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin, c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading, Tel: +44 (0)118 378 2908
Whiteknights, Reading, RG6 6AF, UK Fax: +44 (0)118 975 3094
-------------- next part --------------
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/RDGsuricata.pf560.220713/bin/suricata...done.
[New LWP 11909]
[New LWP 11918]
[New LWP 11903]
[New LWP 11912]
[New LWP 11906]
[New LWP 11921]
[New LWP 11915]
[New LWP 11924]
[New LWP 11927]
[New LWP 11897]
[New LWP 11154]
[New LWP 11900]
[New LWP 11931]
[New LWP 11930]
[New LWP 11932]
[New LWP 11894]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/RDGsuricata/bin/suricata --pfring -c /etc/suricata/suricata-dnacluster.yam'.
Program terminated with signal 11, Segmentation fault.
#0 0x00000000004c2acb in _mm_loadu_si128 (__P=0x7f361205aff8)
at /usr/lib/gcc/x86_64-linux-gnu/4.6/include/emmintrin.h:685
685 return (__m128i) __builtin_ia32_loaddqu ((char const *)__P);
#0 0x00000000004c2acb in _mm_loadu_si128 (__P=0x7f361205aff8)
at /usr/lib/gcc/x86_64-linux-gnu/4.6/include/emmintrin.h:685
No locals.
#1 SCMemcmpLowercase (s1=0x66b2dc, s2=0x7f361205aff8, n=6)
at util-memcmp.h:106
len = 480
b1 = {8286734817810149219, 7596287720686449765}
b2 = {128026086171489, 6}
mask = {32, 0}
r = 0
m = 0
ucase = {23105, 0}
nulls = {0, 0}
uplow = {2314885530818453536, 2314885530818453536}
#2 0x00000000004c3994 in DetectEngineHHDGetBufferForTX (tx=0x7f3601c04d80,
tx_id=1, de_ctx=0x0, det_ctx=0x7f37176e7800, f=0x42486c0,
htp_state=0x7f35f572b870, flags=4 '\004', buffer_len=0x7f36f6fc6a28)
at detect-engine-hhd.c:149
size1 = 6
size2 = 196
headers_buffer = 0x7f35f1e2c600 "Host: platform.twitter.com\r\nConnection: keep-alive\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Ge"...
index = 0
headers = 0x7f35f281c4d0
h = 0x7f360d314f20
headers_buffer_len = 433
i = 7
no_of_headers = 8
__FUNCTION__ = "DetectEngineHHDGetBufferForTX"
#3 0x00000000004c3ffe in DetectEngineRunHttpHeaderMpm (
det_ctx=0x7f37176e7800, f=0x42486c0, htp_state=0x7f35f572b870,
flags=4 '\004', tx=0x7f3601c04d80, idx=1) at detect-engine-hhd.c:195
cnt = 0
buffer_len = 0
buffer = 0x7f360fbbb4c0 "°DÁþ5\177"
#4 0x000000000048e87a in DetectMpmPrefilter (de_ctx=0x483b000,
det_ctx=0x7f37176e7800, smsg=0x7f35e2f58000, p=0x17d1300, flags=4 '\004',
alproto=1, alstate=0x7f35f572b870, sms_runflags=0x7f36f6fc6b8a "\001#")
at detect.c:1016
tx = 0x7f3601c04d80
htp_state = 0x7f35f572b870
tx_progress = 5
idx = 1
total_txs = 2
#5 0x000000000049023e in SigMatchSignatures (th_v=0x6ee67e0,
de_ctx=0x483b000, det_ctx=0x7f37176e7800, p=0x17d1300) at detect.c:1449
sms_runflags = 1 '\001'
alert_flags = 0 '\000'
alproto = 1
idx = 116287456
flags = 4 '\004'
alstate = 0x7f35f572b870
smsg = 0x7f35e2f58000
s = 0x0
sm = 0x0
alversion = 13
reset_de_state = 0
alerts = 0
i = 32567
app_decoder_events = 0
mask = 35 '#'
#6 0x000000000049104c in Detect (tv=0x6ee67e0, p=0x17d1300,
data=0x7f37176e7800, pq=0x7f3737dbd540, postpq=0x0) at detect.c:1850
det_ctx = 0x7f37176e7800
de_ctx = 0x483b000
r = 0
#7 0x00000000005bbb19 in TmThreadsSlotVarRun (tv=0x6ee67e0, p=0x17d1300,
slot=0x7f3737dbd780) at tm-threads.c:542
SlotFunc = 0x490f52 <Detect>
r = TM_ECODE_OK
s = 0x7f3737dbd500
extra_p = 0x7f36f6fc7500
#8 0x0000000000592955 in TmThreadsSlotProcessPkt (tv=0x6ee67e0,
s=0x7f3737dbd780, p=0x17d1300) at tm-threads.h:139
r = TM_ECODE_OK
#9 0x0000000000592fc3 in ReceivePfringLoop (tv=0x6ee67e0, data=0x6411f60,
slot=0x7f3737dbd8c0) at source-pfring.c:323
r = 1
packet_q_len = 4989
ptv = 0x6411f60
p = 0x17d1300
hdr = {ts = {tv_sec = 1374495529, tv_usec = 880201}, caplen = 64,
len = 64, extended_hdr = {timestamp_ns = 18373715326914481920,
rx_direction = 1 '\001', if_index = 6, pkt_hash = 3419002181,
tx = {bounce_interface = -151227008,
reserved = 0xfefc8cbde0e34f00}, parsed_header_len = 0,
parsed_pkt = {dmac = "\000\000\000\000\000",
smac = "\000\000\000\000\000", eth_type = 30080,
vlan_id = 63228, ip_version = 54 '6', l3_proto = 127 '\177',
ip_tos = 0 '\000', ip_src = {v6 = {__in6_u = {
__u6_addr8 = "³\206F\000\000\000\000\000àgî\006\000\000\000", __u6_addr16 = {34483, 70, 0, 0, 26592, 1774, 0, 0}, __u6_addr32 = {
4622003, 0, 116287456, 0}}}, v4 = 4622003}, ip_dst = {
v6 = {__in6_u = {
__u6_addr8 = "\000Oãà½\214üþ\000\000\000\000\000\000\000", __u6_addr16 = {20224, 57571, 36029, 65276, 0, 0, 0, 0}, __u6_addr32 = {
3772993280, 4277963965, 0, 0}}}, v4 = 3772993280},
l4_src_port = 0, l4_dst_port = 0, tcp = {flags = 0 '\000',
seq_num = 4143740320, ack_num = 32566}, tunnel = {
tunnel_id = 4628499, tunneled_proto = 0 '\000',
tunneled_ip_src = {v6 = {__in6_u = {
__u6_addr8 = "àgî\006\000\000\000\000Y¸\027Î;\177\000",
__u6_addr16 = {26592, 1774, 0, 0, 47193, 52759, 32571,
0}, __u6_addr32 = {116287456, 0, 3457661017,
32571}}}, v4 = 116287456}, tunneled_ip_dst = {v6 = {
__in6_u = {
__u6_addr8 = "\000\000\000\000\000\000\000\000\204\227\032Î;\177\000", __u6_addr16 = {0, 0, 0, 0, 38788, 52762, 32571, 0},
__u6_addr32 = {0, 0, 3457849220, 32571}}}, v4 = 0},
tunneled_l4_src_port = 0, tunneled_l4_dst_port = 0},
last_matched_plugin_id = 4, last_matched_rule_id = 0, offset = {
eth_offset = 26592, vlan_offset = 1774, l3_offset = 0,
l4_offset = 0, payload_offset = 30176}}}}
s = 0x7f3737dbd8c0
last_dump = 1374495529
current_time = {tv_sec = 1374495529, tv_usec = 879397}
__FUNCTION__ = "ReceivePfringLoop"
#10 0x00000000005bc450 in TmThreadsSlotPktAcqLoop (td=0x6ee67e0)
at tm-threads.c:682
tv = 0x6ee67e0
s = 0x7f3737dbd8c0
run = 1 '\001'
r = TM_ECODE_OK
slot = 0x0
__FUNCTION__ = "TmThreadsSlotPktAcqLoop"
#11 0x00007f3bce8fde9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#12 0x00007f3bce1afccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#13 0x0000000000000000 in ?? ()
No symbol table info available.
More information about the Oisf-devel
mailing list