[Oisf-devel] Suricata 2.0dev + PF_RING 5.6.0 sporadic crashes in HTPCallbackRequest

Mon Jul 22 16:54:32 UTC 2013

On 07/22/2013 05:38 PM, Victor Julien wrote:
> On 07/22/2013 02:55 PM, Chris Wakelin wrote:
>> On 22/07/13 12:13, Victor Julien wrote:
>>> On 07/22/2013 11:13 AM, Chris Wakelin wrote:
>>>> On 19/07/13 13:58, Anoop Saldanha wrote:
>>>>> On Fri, Jul 19, 2013 at 6:07 PM, Chris Wakelin
>>>>> <c.d.wakelin at reading.ac.uk> wrote:
>>>>>> Hi,
>>>>>>
>>>>>> I recently upgraded our Suricata instances to Suricata 2.0dev (rev
>>>>>> 6229bfa - just a bit before the libhtp unbundling changes) and from
>>>>>> PF_RING 5.5.2 to 5.6.0.
>>>>>>
>>>>>> We're getting sporadic crashes in both sensors; they can go for a day
>>>>>> without crashing, then crash three times in half an hour, so it looks
>>>>>> like it's triggered by some very specific traffic.
>>>>>>
>>>>>> Looking in the backtrace, the ReceivePfringLoop frame suggests to me the
>>>>>> packet is corrupt (as far as I can tell - e.g. src and dst port are 0
>>>>>> and protocol 127, IPv4 addresses don't look like our local ones).
>>>>>>
>>>>>>> #17 0x0000000000592c0b in ReceivePfringLoop (tv=0xc971540, data=0x7fa92cd51f00, slot=0x86ca8c0) at source-pfring.c:311
>>>>>>>         r = 1
>>>>>>>         packet_q_len = 4989
>>>>>>>         ptv = 0x7fa92cd51f00
>>>>>>>         p = 0x420d400
>>>>>>>         hdr = {ts = {tv_sec = 1374235900, tv_usec = 231639}, caplen = 60, len = 60, extended_hdr = {timestamp_ns = 4314826383835889664, rx_direction = 1 '\001', if_index = 6, pkt_hash = 1191440499, tx = {bounce_interface = 764450176, reserved = 0x3be157bc34058000},
>>>>>>>             parsed_header_len = 0, parsed_pkt = {dmac = "\000\000\000\000\000", smac = "\000\000\000\000\000", eth_type = 38272, vlan_id = 11664, ip_version = 166 'ï¿½', l3_proto = 127 '\177', ip_tos = 0 '\000', ip_src = {v6 = {__in6_u = {
>>>>>>>                     __u6_addr8 = "ï¿½}F\000\000\000\000\000@\025\227\f\000\000\000", __u6_addr16 = {32243, 70, 0, 0, 5440, 3223, 0, 0}, __u6_addr32 = {4619763, 0, 211227968, 0}}}, v4 = 4619763}, ip_dst = {v6 = {__in6_u = {
>>>>>>>                     __u6_addr8 = "\000\200\005\064ï¿½Wï¿½;\000\000\000\000\000\000\000", __u6_addr16 = {32768, 13317, 22460, 15329, 0, 0, 0, 0}, __u6_addr32 = {872775680, 1004623804, 0, 0}}}, v4 = 872775680}, l4_src_port = 0, l4_dst_port = 0, tcp = {flags = 0 '\000',
>>>>>>>                 seq_num = 764450208, ack_num = 32678}, tunnel = {tunnel_id = 4626108, tunneled_proto = 0 '\000', tunneled_ip_src = {v6 = {__in6_u = {__u6_addr8 = "@\025\227\f\000\000\000\000Y()Þª\177\000", __u6_addr16 = {5440, 3223, 0, 0, 10329, 56873, 32682, 0},
>>>>>>>                       __u6_addr32 = {211227968, 0, 3727239257, 32682}}}, v4 = 211227968}, tunneled_ip_dst = {v6 = {__in6_u = {__u6_addr8 = "\000\000\000\000\000\000\000\000\204\a,Þª\177\000", __u6_addr16 = {0, 0, 0, 0, 1924, 56876, 32682, 0}, __u6_addr32 = {0, 0,
>>>>>>>                         3727427460, 32682}}}, v4 = 0}, tunneled_l4_src_port = 0, tunneled_l4_dst_port = 0}, last_matched_plugin_id = 4, last_matched_rule_id = 0, offset = {eth_offset = 5440, vlan_offset = 3223, l3_offset = 0, l4_offset = 0, payload_offset = -27168}}}}
>>>>>>>         s = 0x86ca8c0
>>>>>>>         last_dump = 1374235900
>>>>>>>         current_time = {tv_sec = 1374235900, tv_usec = 231263}
>>>>>>>         __FUNCTION__ = "ReceivePfringLoop"
>>>>>>
>>>>>> I've attached a backtrace from a core that was generated a few minutes
>>>>>> ago (Suricata was compiled with CFLAGS="-ggdb -O0").
>>>>>>
>>>>>> Any ideas what traffic caused this? (My feeling is the corrupt packets,
>>>>>> if that's what they are, are probably PF_RING's fault, but of course
>>>>>> Suricata shouldn't crash even then.)
>>>>>>
>>>>>> I can downgrade Suricata, but alas I'm not allowed to touch PF_RING
>>>>>> without going through a Change Control process (it upset the border
>>>>>> switch once).
>>>>>>
>>>>>
>>>>> Can you run the lastest master(post 0.5.x changes).  There were some
>>>>> bugs in libhtp which were fixed explicitly for 1.4.x, and for the
>>>>> master we relied on the 0.5.x fixing it.
>>>>>
>>>>
>>>> Still crashing sporadically I'm afraid, but now it's mostly in
>>>> htp_validate_hostname. I've attached another backtrace - does the frame
>>>> in ReceivePfringLoop make any sense?
>>>
>>> I agree the pfring data looks weird. Might be uninitialized, maybe
>>> pfring doesn't init it if it doesn't use it. Despite this value, suri
>>> figured out it's TCP.
>>>
>>
>> I did as Ivan suggested and upgraded to latest libhtp git (I was running
>> master from Friday 19th July) and latest Suricata too (likewise)
>>
>> Here's a backtrace for another crash we've been seeing a bit less
>> frequently but has just re-occurred (with the latest git releases).
>>
>> Again the ReceivePfringLoop looks a bit suspect to me :)
> 
> I've pinged Luca about it, will let you know.
> 

I was told:

"in order to optimise the computation the pfring header does not contain
parsing info by default:
- with standard (kernel-based) drivers you should add
PF_RING_LONG_HEADER to the pfring_open() flags.
- with dna we parse the packet in 1-copy mode only (passing a buffer
with a buffer-len > 0 to pfring_recv()) for the same reason.
You can explicitly call the parsing routine calling pfring_parse_pkt()"

So the seemingly random/weird data is the result of us not using
PF_RING_LONG_HEADER and pfring_parse_pkt(). So it's not an indication of
something bad.

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------