[Oisf-devel] Suricata 2.0dev + PF_RING 5.6.0 sporadic crashes in HTPCallbackRequest
Chris Wakelin
c.d.wakelin at reading.ac.uk
Mon Jul 22 05:13:16 EDT 2013
On 19/07/13 13:58, Anoop Saldanha wrote:
> On Fri, Jul 19, 2013 at 6:07 PM, Chris Wakelin
> <c.d.wakelin at reading.ac.uk> wrote:
>> Hi,
>>
>> I recently upgraded our Suricata instances to Suricata 2.0dev (rev
>> 6229bfa - just a bit before the libhtp unbundling changes) and from
>> PF_RING 5.5.2 to 5.6.0.
>>
>> We're getting sporadic crashes in both sensors; they can go for a day
>> without crashing, then crash three times in half an hour, so it looks
>> like it's triggered by some very specific traffic.
>>
>> Looking in the backtrace, the ReceivePfringLoop frame suggests to me the
>> packet is corrupt (as far as I can tell - e.g. src and dst port are 0
>> and protocol 127, IPv4 addresses don't look like our local ones).
>>
>>> #17 0x0000000000592c0b in ReceivePfringLoop (tv=0xc971540, data=0x7fa92cd51f00, slot=0x86ca8c0) at source-pfring.c:311
>>> r = 1
>>> packet_q_len = 4989
>>> ptv = 0x7fa92cd51f00
>>> p = 0x420d400
>>> hdr = {ts = {tv_sec = 1374235900, tv_usec = 231639}, caplen = 60, len = 60, extended_hdr = {timestamp_ns = 4314826383835889664, rx_direction = 1 '\001', if_index = 6, pkt_hash = 1191440499, tx = {bounce_interface = 764450176, reserved = 0x3be157bc34058000},
>>> parsed_header_len = 0, parsed_pkt = {dmac = "\000\000\000\000\000", smac = "\000\000\000\000\000", eth_type = 38272, vlan_id = 11664, ip_version = 166 '�', l3_proto = 127 '\177', ip_tos = 0 '\000', ip_src = {v6 = {__in6_u = {
>>> __u6_addr8 = "�}F\000\000\000\000\000@\025\227\f\000\000\000", __u6_addr16 = {32243, 70, 0, 0, 5440, 3223, 0, 0}, __u6_addr32 = {4619763, 0, 211227968, 0}}}, v4 = 4619763}, ip_dst = {v6 = {__in6_u = {
>>> __u6_addr8 = "\000\200\005\064�W�;\000\000\000\000\000\000\000", __u6_addr16 = {32768, 13317, 22460, 15329, 0, 0, 0, 0}, __u6_addr32 = {872775680, 1004623804, 0, 0}}}, v4 = 872775680}, l4_src_port = 0, l4_dst_port = 0, tcp = {flags = 0 '\000',
>>> seq_num = 764450208, ack_num = 32678}, tunnel = {tunnel_id = 4626108, tunneled_proto = 0 '\000', tunneled_ip_src = {v6 = {__in6_u = {__u6_addr8 = "@\025\227\f\000\000\000\000Y()Þª\177\000", __u6_addr16 = {5440, 3223, 0, 0, 10329, 56873, 32682, 0},
>>> __u6_addr32 = {211227968, 0, 3727239257, 32682}}}, v4 = 211227968}, tunneled_ip_dst = {v6 = {__in6_u = {__u6_addr8 = "\000\000\000\000\000\000\000\000\204\a,Þª\177\000", __u6_addr16 = {0, 0, 0, 0, 1924, 56876, 32682, 0}, __u6_addr32 = {0, 0,
>>> 3727427460, 32682}}}, v4 = 0}, tunneled_l4_src_port = 0, tunneled_l4_dst_port = 0}, last_matched_plugin_id = 4, last_matched_rule_id = 0, offset = {eth_offset = 5440, vlan_offset = 3223, l3_offset = 0, l4_offset = 0, payload_offset = -27168}}}}
>>> s = 0x86ca8c0
>>> last_dump = 1374235900
>>> current_time = {tv_sec = 1374235900, tv_usec = 231263}
>>> __FUNCTION__ = "ReceivePfringLoop"
>>
>> I've attached a backtrace from a core that was generated a few minutes
>> ago (Suricata was compiled with CFLAGS="-ggdb -O0").
>>
>> Any ideas what traffic caused this? (My feeling is the corrupt packets,
>> if that's what they are, are probably PF_RING's fault, but of course
>> Suricata shouldn't crash even then.)
>>
>> I can downgrade Suricata, but alas I'm not allowed to touch PF_RING
>> without going through a Change Control process (it upset the border
>> switch once).
>>
>
> Can you run the lastest master(post 0.5.x changes). There were some
> bugs in libhtp which were fixed explicitly for 1.4.x, and for the
> master we relied on the 0.5.x fixing it.
>
Still crashing sporadically I'm afraid, but now it's mostly in
htp_validate_hostname. I've attached another backtrace - does the frame
in ReceivePfringLoop make any sense?
Best Wishes,
Chris
--
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin, c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading, Tel: +44 (0)118 378 8439
Whiteknights, Reading, RG6 2AF, UK Fax: +44 (0)118 975 3094
-------------- next part --------------
GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
For bug reporting instructions, please see:
<http://bugs.launchpad.net/gdb-linaro/>...
Reading symbols from /opt/RDGsuricata.pf560.190713/bin/suricata...done.
[New LWP 30544]
[New LWP 30550]
[New LWP 30541]
[New LWP 30547]
[New LWP 30553]
[New LWP 30559]
[New LWP 30556]
[New LWP 30562]
[New LWP 30565]
[New LWP 30535]
[New LWP 30538]
[New LWP 30532]
[New LWP 30570]
[New LWP 30569]
[New LWP 29894]
[New LWP 30568]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/opt/RDGsuricata/bin/suricata --pfring -c /etc/suricata/suricata-dnacluster.yam'.
Program terminated with signal 11, Segmentation fault.
#0 htp_validate_hostname (hostname=0x0) at htp_util.c:2468
#0 htp_validate_hostname (hostname=0x0) at htp_util.c:2468
data = 0x7fcee6b13fb8 "Host"
len = <optimised out>
pos = <optimised out>
#1 0x00007fd4b3ed3a08 in htp_parse_header_hostport (
hostport=<optimised out>, hostname=0x7fcfece2cec0, port=<optimised out>,
flags=0x7fcee6b96a70) at htp_util.c:597
invalid = 1
rc = 1
#2 0x00007fd4b3ecf665 in htp_tx_process_request_headers (tx=0x7fcee6b96900)
at htp_transaction.c:478
hostname = <optimised out>
port = <optimised out>
rc = 1
cl = 0x7fcee6b13f20
te = <optimised out>
h = <optimised out>
ct = <optimised out>
#3 htp_tx_state_request_headers (tx=0x7fcee6b96900) at htp_transaction.c:901
rc = 1
#4 0x00007fd4b3ecb885 in htp_connp_REQ_HEADERS (connp=0x7fcee6c73c40)
at htp_request.c:583
data = 0x7fcfece2da97 "\r\nNGfZND00bDAHaTE3MZY1MGQ1bGM0zjJjMWJHbDrIYWU0aZc1aTBZbDQZYjAmJmzpOGVZNXplPTc1MZU1bDimdmVyPTEmzm4yOWf0PWpZO29mc243cmalPXHmzGxFcXVlcnkmzmlszW5HOWU4rfaDMDQxaDQlMkVKUEcmdWluPTkxMjk5aTc3MyzjdXa0O23pzD0xMj"...
len = 2
#5 0x00007fd4b3ecbcda in htp_connp_req_data (connp=0x7fcee6c73c40,
timestamp=<optimised out>, data=<optimised out>, len=<optimised out>)
at htp_request.c:851
rc = <optimised out>
#6 0x0000000000429dc4 in HTPHandleRequestData (f=0x378ac60,
htp_state=0x6a83960, pstate=0x7fcee6a17138,
input=0x7fcfece2d9d0 "POST /qxf/check_download HTTP/1.1\r\nAccept: */*\r\nHost: \r\nContent-Type: applation/octet-stream;\r\nContent-Transfe-Encoding: BINARY\r\nContent-Length: 208\r\nConnection: Keep-Alive\r\nCache-Control: no-cache\r\n\r"..., input_len=409, local_data=0x0, output=0x7fcfece2d830)
at app-layer-htp.c:642
r = -1
ret = 1
hstate = 0x6a83960
__FUNCTION__ = "HTPHandleRequestData"
__PRETTY_FUNCTION__ = "HTPHandleRequestData"
ts = {tv_sec = 1374478827, tv_usec = 0}
#7 0x000000000043b9c3 in AppLayerDoParse (local_data=0x0, f=0x378ac60,
app_layer_state=0x6a83960, parser_state=0x7fcee6a17138,
input=0x7fcfece2d9d0 "POST /qxf/check_download HTTP/1.1\r\nAccept: */*\r\nHost: \r\nContent-Type: applation/octet-stream;\r\nContent-Transfe-Encoding: BINARY\r\nContent-Length: 208\r\nConnection: Keep-Alive\r\nCache-Control: no-cache\r\n\r"..., input_len=409, parser_idx=1, proto=1) at app-layer-parser.c:887
retval = 0
result = {head = 0x0, tail = 0x0, cnt = 0}
r = -1
e = 0x501767d193889900
#8 0x000000000043be5e in AppLayerParse (local_data=0x0, f=0x378ac60,
proto=1 '\001', flags=5 '\005',
input=0x7fcfece2d9d0 "POST /qxf/check_download HTTP/1.1\r\nAccept: */*\r\nHost: \r\nContent-Type: applation/octet-stream;\r\nContent-Transfe-Encoding: BINARY\r\nContent-Length: 208\r\nConnection: Keep-Alive\r\nCache-Control: no-cache\r\n\r"..., input_len=409) at app-layer-parser.c:1093
r = -1274108377
parser_idx = 1
p = 0x8f5840
ssn = 0x7fcee77faf00
parser_state_store = 0x7fcee6a17120
parser_state = 0x7fcee6a17138
app_layer_state = 0x6a83960
#9 0x0000000000413546 in AppLayerHandleTCPData (dp_ctx=0x86c4608,
f=0x378ac60, ssn=0x7fcee77faf00,
data=0x7fcfece2d9d0 "POST /qxf/check_download HTTP/1.1\r\nAccept: */*\r\nHost: \r\nContent-Type: applation/octet-stream;\r\nContent-Transfe-Encoding: BINARY\r\nContent-Length: 208\r\nConnection: Keep-Alive\r\nCache-Control: no-cache\r\n\r"..., data_len=409, flags=5 '\005') at app-layer.c:161
r = 0
#10 0x00000000005ab301 in StreamTcpReassembleAppLayer (tv=0x1832f460,
ra_ctx=0x86c4600, ssn=0x7fcee77faf00, stream=0x7fcee77faf50, p=0x16a9400)
at stream-tcp-reassemble.c:2933
flags = 5 '\005'
seg_tail = 0x62a6930
ra_base_seq = 2828352677
data = "POST /qxf/check_download HTTP/1.1\r\nAccept: */*\r\nHost: \r\nContent-Type: applation/octet-stream;\r\nContent-Transfe-Encoding: BINARY\r\nContent-Length: 208\r\nConnection: Keep-Alive\r\nCache-Control: no-cache\r\n\r"...
data_len = 409
payload_offset = 0
payload_len = 409
next_seq = 2828352678
seg = 0x0
__PRETTY_FUNCTION__ = "StreamTcpReassembleAppLayer"
#11 0x00000000005abbca in StreamTcpReassembleHandleSegmentUpdateACK (
tv=0x1832f460, ra_ctx=0x86c4600, ssn=0x7fcee77faf00,
stream=0x7fcee77faf50, p=0x16a9400) at stream-tcp-reassemble.c:3295
r = 0
#12 0x00000000005abd29 in StreamTcpReassembleHandleSegment (tv=0x1832f460,
ra_ctx=0x86c4600, ssn=0x7fcee77faf00, stream=0x7fcee77faf08, p=0x16a9400,
pq=0x6dcbab0) at stream-tcp-reassemble.c:3369
opposing_stream = 0x7fcee77faf50
#13 0x000000000059bf10 in HandleEstablishedPacketToClient (tv=0x1832f460,
ssn=0x7fcee77faf00, p=0x16a9400, stt=0x6dcbaa0, pq=0x6dcbab0)
at stream-tcp.c:2048
zerowindowprobe = 0
#14 0x000000000059c8e7 in StreamTcpPacketStateEstablished (tv=0x1832f460,
p=0x16a9400, stt=0x6dcbaa0, ssn=0x7fcee77faf00, pq=0x6dcbab0)
at stream-tcp.c:2294
No locals.
#15 0x00000000005a263b in StreamTcpPacket (tv=0x1832f460, p=0x16a9400,
stt=0x6dcbaa0, pq=0x7fd02dd73680) at stream-tcp.c:4200
ssn = 0x7fcee77faf00
#16 0x00000000005a2e52 in StreamTcp (tv=0x1832f460, p=0x16a9400,
data=0x6dcbaa0, pq=0x7fd02dd73680, postpq=0x0) at stream-tcp.c:4441
stt = 0x6dcbaa0
ret = TM_ECODE_OK
#17 0x00000000005bbfe5 in TmThreadsSlotVarRun (tv=0x1832f460, p=0x16a9400,
slot=0x7fd02dd73780) at tm-threads.c:542
SlotFunc = 0x5a2d49 <StreamTcp>
r = TM_ECODE_OK
s = 0x7fd02dd73640
extra_p = 0x7fcfece2f500
#18 0x0000000000592e4d in TmThreadsSlotProcessPkt (tv=0x1832f460,
s=0x7fd02dd73780, p=0x16a9400) at tm-threads.h:139
r = TM_ECODE_OK
#19 0x00000000005934bb in ReceivePfringLoop (tv=0x1832f460,
data=0x7fd08493c1e0, slot=0x7fd02dd738c0) at source-pfring.c:323
r = 1
packet_q_len = 4989
ptv = 0x7fd08493c1e0
p = 0x16a9400
hdr = {ts = {tv_sec = 1374478827, tv_usec = 390403}, caplen = 60,
len = 60, extended_hdr = {timestamp_ns = 5771195597319608576,
rx_direction = 1 '\001', if_index = 6, pkt_hash = 3252505864,
tx = {bounce_interface = -320670336,
reserved = 0x501767d193889900}, parsed_header_len = 0,
parsed_pkt = {dmac = "\000\000\000\000\000",
smac = "\000\000\000\000\000", eth_type = 62848,
vlan_id = 60642, ip_version = 207 'Ï', l3_proto = 127 '\177',
ip_tos = 0 '\000', ip_src = {v6 = {__in6_u = {
__u6_addr8 = "³\206F\000\000\000\000\000`ô2\030\000\000\000", __u6_addr16 = {34483, 70, 0, 0, 62560, 6194, 0, 0}, __u6_addr32 = {
4622003, 0, 405992544, 0}}}, v4 = 4622003}, ip_dst = {
v6 = {__in6_u = {
__u6_addr8 = "\000\231\210\223Ñg\027P\000\000\000\000\000\000\000", __u6_addr16 = {39168, 37768, 26577, 20503, 0, 0, 0, 0},
__u6_addr32 = {2475202816, 1343711185, 0, 0}}},
v4 = 2475202816}, l4_src_port = 0, l4_dst_port = 0, tcp = {
flags = 0 '\000', seq_num = 3974296992, ack_num = 32719},
tunnel = {tunnel_id = 4628499, tunneled_proto = 0 '\000',
tunneled_ip_src = {v6 = {__in6_u = {
__u6_addr8 = "`ô2\030\000\000\000\000Y\b£²Ô\177\000",
__u6_addr16 = {62560, 6194, 0, 0, 2137, 45731, 32724,
0}, __u6_addr32 = {405992544, 0, 2997028953,
32724}}}, v4 = 405992544}, tunneled_ip_dst = {v6 = {
__in6_u = {
__u6_addr8 = "\000\000\000\000\000\000\000\000\204祲Ô\177\000", __u6_addr16 = {0, 0, 0, 0, 59268, 45733, 32724, 0}, __u6_addr32 = {0,
0, 2997217156, 32724}}}, v4 = 0},
tunneled_l4_src_port = 0, tunneled_l4_dst_port = 0},
last_matched_plugin_id = 4, last_matched_rule_id = 0, offset = {
eth_offset = -2976, vlan_offset = 6194, l3_offset = 0,
l4_offset = 0, payload_offset = -2592}}}}
s = 0x7fd02dd738c0
last_dump = 1374478827
current_time = {tv_sec = 1374478827, tv_usec = 390402}
__FUNCTION__ = "ReceivePfringLoop"
#20 0x00000000005bc91c in TmThreadsSlotPktAcqLoop (td=0x1832f460)
at tm-threads.c:682
tv = 0x1832f460
s = 0x7fd02dd738c0
run = 1 '\001'
r = TM_ECODE_OK
slot = 0x0
__FUNCTION__ = "TmThreadsSlotPktAcqLoop"
#21 0x00007fd4b31b2e9a in start_thread ()
from /lib/x86_64-linux-gnu/libpthread.so.0
No symbol table info available.
#22 0x00007fd4b2a64ccd in clone () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#23 0x0000000000000000 in ?? ()
No symbol table info available.
More information about the Oisf-devel
mailing list