[Oisf-devel] Packet Array in 1.0.x

Tue Sep 7 17:01:24 UTC 2010

Victor,

Just to be clear, this problem is occurring in the Suricata code base
and not within the Bivio modifications. The code Im testing this with is
unmodified. 

// Joel 

>-----Original Message-----
>From: Victor Julien [mailto:victor at inliniac.net]
>Sent: Monday, September 06, 2010 7:55 AM
>To: Joel Ebrahimi
>Cc: oisf-devel at openinfosecfoundation.org; Will Metcalf
>Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Joel Ebrahimi wrote:
>> Hey Victor,
>>
>> In the first call to pcap_dispatch, when this statement:
>>
>>     (pcap_max_read_packets < packet_q_len) ?  *pcap_max_read_packets
:
>> packet_q_len
>>
>> pcap_max_read_packet is 0, so 0 is passed in to cnt for
pcap_dispatch. A
>> 0 or a -1 here causes the all the packets received in the buffer to
be
>> processed.
>
>Right. In source-pcap.c we make sure we never get to that condition by
>entering a wait loop that sets packet_q_len based on the number of
>available packets in the packet pool. As long as it's 0 it will remain
>in that loop.
>
>Did you implement a similar protection?
>
>> In my test cast I can see PcapCallback called 10's or thousands or
>> times. It appear that depending on the pcap file Im sending that
either
>> memory is exhausted here in PcapCallback and the process is killed or
if
>> it returns then the variable ptv->array_idx is set to something
>> extremely high (like 15000) and when the for loop happens after the
>> pcap_dispatch, then  once cnt goes passed 256 we are accessing
outside
>> our memory space and I get the segfault.
>
>Problem is that it will be hard to set the array size to be big enough.
>  So I think choosing and then enforcing a limit is inevitable.
>
>Cheers,
>Victor
>
>>
>> // Joel
>>
>>> -----Original Message-----
>>> From: Victor Julien [mailto:victor at inliniac.net]
>>> Sent: Monday, August 23, 2010 2:01 AM
>>> To: Joel Ebrahimi
>>> Cc: Will Metcalf; oisf-devel at openinfosecfoundation.org
>>> Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>>
>> Hi Joel,
>>
>> For the pcap module we pass a variable to pcap_dispatch that makes
sure
>> we read a max number of packets equal to or lower than 256, based on
>>> the
>> amount of available packets in the PacketPool. If none are available
we
>> wait. See lines 180 to 200 in source-pcap.c
>>
>> Cheers,
>> Victor
>>
>> Joel Ebrahimi wrote:
>>>>> Sure. As I said though its not always a segfault but I can see all
>>> the
>>>>> memory being used up on the system. Here is the segfault
backtrace:
>>>>>
>>>>> #0  0x10012150 in PacketEnqueue (q=0x1080eb70, p=0x2a9) at
>>>>> packet-queue.c:142
>>>>> #1  0x10012f98 in ReceivePcap (tv=0x1080ea78, p=0x10265f70,
>>>>> data=0x107faef8, pq=0x1080eb10, postpq=0x1080eb70)
>>>>>     at source-pcap.c:219
>>>>> #2  0x10104f44 in TmThreadsSlot1 (td=0x1080ea78) at
tm-threads.c:371
>>>>> #3  0x0ff0d994 in start_thread () from /lib/libpthread.so.0
>>>>> #4  0x0ff0d994 in start_thread () from /lib/libpthread.so.0
>>>>> Previous frame inner to this frame (corrupt stack?)
>>>>>
>>>>> Really the problem seems to get ugly in the ReceivePcap function
>>> here:
>>>>>     for (cnt = 0; cnt < ptv->array_idx; cnt++) {
>>>>>         Packet *pp = ptv->array[cnt];
>>>>>
>>>>>         /* enqueue all but the first in the postpq, the first
>>>>>          * pkt is handled by the tv "out handler" */
>>>>>         if (cnt > 0) {
>>>>>             PacketEnqueue(postpq, pp);
>>>>>         }
>>>>>     }
>>>>>
>>>>> The problem being that array_idx keeps incrementing. So in that
>>> specific
>>>>> backtrace case above array_idx is 6538 and when this section of
code
>>> is
>>>>> hit, once count exceeds 255 we are outside the range of memory and
as
>>>>> you can see from the backtrace that in the function PacketEnqueue
>>> where
>>>>> p's address is wrong.
>>>>>
>>>>> I know the array stuff is new in the 1.0.x release and looking it
>>> over I
>>>>> don't see a case that ensures it stays within the max 256
allocated
>>> or I
>>>>> don't see a case where array_idx ever gets reset.
>>>>>
>>>>> // Joel
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Will Metcalf [mailto:william.metcalf at gmail.com]
>>>>>> Sent: Thursday, August 19, 2010 4:36 PM
>>>>>> To: Joel Ebrahimi
>>>>>> Cc: oisf-devel at openinfosecfoundation.org
>>>>>> Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>>>>>
>>>>>> Can you give us the output of a backtrace Joel?  You can send
>>> off-list
>>>>>> if you so desire.
>>>>>>
>>>>>> Regards,
>>>>>>
>>>>>> Will
>>>>>>
>>>>>> On Thu, Aug 19, 2010 at 5:19 PM, Joel Ebrahimi
<jebrahimi at bivio.net>
>>>>> wrote:
>>>>>>> I'm seeing multiple problems with the array that was added to
>>>>>>> PcapThreadVars_.
>>>>>>>
>>>>>>> I either exhaust memory completely or segfault. I think they
both
>>>>> stem
>>>>>>> from the same issue with is that the array seems to keep filling
>>> and
>>>>>>> eventually there is an access outside of PCAP_FILE_MAX_PKTS
(256)
>>>>> which
>>>>>>> causes the segfaults.
>>>>>>>
>>>>>>> Is the array supposed to hold a finite number of past packets
ie.
>>> 256
>>>>> or
>>>>>>> whatever PCAP_FILE_MAX_PKTS is set to? If so then I suppose a
>>> modulus
>>>>>>> operator or some kind of reset is needed.
>>>>>>>
>>>>>>> // Joel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Joel Ebrahimi
>>>>>>> Solutions Architect
>>>>>>> Bivio Networks
>>>>>>> jebrahimi at bivio.net
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Oisf-devel mailing list
>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>
>
>- --
>- ---------------------------------------------
>Victor Julien
>http://www.inliniac.net/
>PGP: http://www.inliniac.net/victorjulien.asc
>- ---------------------------------------------
>
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG v1.4.9 (GNU/Linux)
>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
>iEYEARECAAYFAkyFAMkACgkQiSMBBAuniMd6iQCeMytxiSc1DigDWNQd9qH+BVC0
>GMwAn3TfL3rN8uvEGJZcXo3fU2kY0wk+
>=dBGn
>-----END PGP SIGNATURE-----