[Oisf-devel] Suricata 2.0dev + PF_RING 5.6.0 sporadic crashes in HTPCallbackRequest

[Victor Julien]

On 07/22/2013 11:13 AM, Chris Wakelin wrote:
> On 19/07/13 13:58, Anoop Saldanha wrote:
>> On Fri, Jul 19, 2013 at 6:07 PM, Chris Wakelin
>> <c.d.wakelin at reading.ac.uk> wrote:
>>> Hi,
>>>
>>> I recently upgraded our Suricata instances to Suricata 2.0dev (rev
>>> 6229bfa - just a bit before the libhtp unbundling changes) and from
>>> PF_RING 5.5.2 to 5.6.0.
>>>
>>> We're getting sporadic crashes in both sensors; they can go for a day
>>> without crashing, then crash three times in half an hour, so it looks
>>> like it's triggered by some very specific traffic.
>>>
>>> Looking in the backtrace, the ReceivePfringLoop frame suggests to me the
>>> packet is corrupt (as far as I can tell - e.g. src and dst port are 0
>>> and protocol 127, IPv4 addresses don't look like our local ones).
>>>
>>>> #17 0x0000000000592c0b in ReceivePfringLoop (tv=0xc971540, data=0x7fa92cd51f00, slot=0x86ca8c0) at source-pfring.c:311
>>>>         r = 1
>>>>         packet_q_len = 4989
>>>>         ptv = 0x7fa92cd51f00
>>>>         p = 0x420d400
>>>>         hdr = {ts = {tv_sec = 1374235900, tv_usec = 231639}, caplen = 60, len = 60, extended_hdr = {timestamp_ns = 4314826383835889664, rx_direction = 1 '\001', if_index = 6, pkt_hash = 1191440499, tx = {bounce_interface = 764450176, reserved = 0x3be157bc34058000},
>>>>             parsed_header_len = 0, parsed_pkt = {dmac = "\000\000\000\000\000", smac = "\000\000\000\000\000", eth_type = 38272, vlan_id = 11664, ip_version = 166 '�', l3_proto = 127 '\177', ip_tos = 0 '\000', ip_src = {v6 = {__in6_u = {
>>>>                     __u6_addr8 = "�}F\000\000\000\000\000@\025\227\f\000\000\000", __u6_addr16 = {32243, 70, 0, 0, 5440, 3223, 0, 0}, __u6_addr32 = {4619763, 0, 211227968, 0}}}, v4 = 4619763}, ip_dst = {v6 = {__in6_u = {
>>>>                     __u6_addr8 = "\000\200\005\064�W�;\000\000\000\000\000\000\000", __u6_addr16 = {32768, 13317, 22460, 15329, 0, 0, 0, 0}, __u6_addr32 = {872775680, 1004623804, 0, 0}}}, v4 = 872775680}, l4_src_port = 0, l4_dst_port = 0, tcp = {flags = 0 '\000',
>>>>                 seq_num = 764450208, ack_num = 32678}, tunnel = {tunnel_id = 4626108, tunneled_proto = 0 '\000', tunneled_ip_src = {v6 = {__in6_u = {__u6_addr8 = "@\025\227\f\000\000\000\000Y()ު\177\000", __u6_addr16 = {5440, 3223, 0, 0, 10329, 56873, 32682, 0},
>>>>                       __u6_addr32 = {211227968, 0, 3727239257, 32682}}}, v4 = 211227968}, tunneled_ip_dst = {v6 = {__in6_u = {__u6_addr8 = "\000\000\000\000\000\000\000\000\204\a,ު\177\000", __u6_addr16 = {0, 0, 0, 0, 1924, 56876, 32682, 0}, __u6_addr32 = {0, 0,
>>>>                         3727427460, 32682}}}, v4 = 0}, tunneled_l4_src_port = 0, tunneled_l4_dst_port = 0}, last_matched_plugin_id = 4, last_matched_rule_id = 0, offset = {eth_offset = 5440, vlan_offset = 3223, l3_offset = 0, l4_offset = 0, payload_offset = -27168}}}}
>>>>         s = 0x86ca8c0
>>>>         last_dump = 1374235900
>>>>         current_time = {tv_sec = 1374235900, tv_usec = 231263}
>>>>         __FUNCTION__ = "ReceivePfringLoop"
>>>
>>> I've attached a backtrace from a core that was generated a few minutes
>>> ago (Suricata was compiled with CFLAGS="-ggdb -O0").
>>>
>>> Any ideas what traffic caused this? (My feeling is the corrupt packets,
>>> if that's what they are, are probably PF_RING's fault, but of course
>>> Suricata shouldn't crash even then.)
>>>
>>> I can downgrade Suricata, but alas I'm not allowed to touch PF_RING
>>> without going through a Change Control process (it upset the border
>>> switch once).
>>>
>>
>> Can you run the lastest master(post 0.5.x changes).  There were some
>> bugs in libhtp which were fixed explicitly for 1.4.x, and for the
>> master we relied on the 0.5.x fixing it.
>>
> 
> Still crashing sporadically I'm afraid, but now it's mostly in
> htp_validate_hostname. I've attached another backtrace - does the frame
> in ReceivePfringLoop make any sense?

I agree the pfring data looks weird. Might be uninitialized, maybe
pfring doesn't init it if it doesn't use it. Despite this value, suri
figured out it's TCP.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------