[Oisf-devel] Packet Array in 1.0.x

Joel Ebrahimi jebrahimi at bivio.net
Tue Sep 7 20:34:13 UTC 2010


I have looked over the code and I don't see any checks that help. Let me
explain the 2 problems I see.

First when pcap_dispatch is called I have a large number of packets that
are available. So this in turn calls PcapCallback. Now something
PcapCallback is called 15,000 times in a row which will instantiate a
lot of memory with the calls to PacketGetFromQueueOrAlloc(), so in this
case I exhaust the entire memory of the system creating buffers for
15,000 packets. So this results in Linux killing the process.

In the other case I do not exhaust all the memory, lets say PcapCallback
is only called 12,000 times. Now in this case I return from the callback
and move into the for loop. So at this point array_idx is 12,000. So
when I enter the for loop cnt is then incremented up to 12,000. The
problem here though is that the ptv->array only has 256 slots allocated
for it, so since cnt exceeds 256 and there is an access to the variable
ptv->array[cnt]; this will cause a segfault.

Does that explain whats happening a little clearer? Im not sure why no
one else has seen this yet, unless their buffers are small and the
PcapCallback routine is called less than 256 times before processing.

// Joel 
>-----Original Message-----
>From: oisf-devel-bounces at openinfosecfoundation.org [mailto:oisf-devel-
>bounces at openinfosecfoundation.org] On Behalf Of Joel Ebrahimi
>Sent: Tuesday, September 07, 2010 10:01 AM
>To: Victor Julien
>Cc: oisf-devel at openinfosecfoundation.org
>Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>Just to be clear, this problem is occurring in the Suricata code base
>and not within the Bivio modifications. The code Im testing this with
>// Joel
>>-----Original Message-----
>>From: Victor Julien [mailto:victor at inliniac.net]
>>Sent: Monday, September 06, 2010 7:55 AM
>>To: Joel Ebrahimi
>>Cc: oisf-devel at openinfosecfoundation.org; Will Metcalf
>>Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>Hash: SHA1
>>Joel Ebrahimi wrote:
>>> Hey Victor,
>>> In the first call to pcap_dispatch, when this statement:
>>>     (pcap_max_read_packets < packet_q_len) ?  *pcap_max_read_packets
>>> packet_q_len
>>> pcap_max_read_packet is 0, so 0 is passed in to cnt for
>pcap_dispatch. A
>>> 0 or a -1 here causes the all the packets received in the buffer to
>>> processed.
>>Right. In source-pcap.c we make sure we never get to that condition by
>>entering a wait loop that sets packet_q_len based on the number of
>>available packets in the packet pool. As long as it's 0 it will remain
>>in that loop.
>>Did you implement a similar protection?
>>> In my test cast I can see PcapCallback called 10's or thousands or
>>> times. It appear that depending on the pcap file Im sending that
>>> memory is exhausted here in PcapCallback and the process is killed
>>> it returns then the variable ptv->array_idx is set to something
>>> extremely high (like 15000) and when the for loop happens after the
>>> pcap_dispatch, then  once cnt goes passed 256 we are accessing
>>> our memory space and I get the segfault.
>>Problem is that it will be hard to set the array size to be big
>>  So I think choosing and then enforcing a limit is inevitable.
>>> // Joel
>>>> -----Original Message-----
>>>> From: Victor Julien [mailto:victor at inliniac.net]
>>>> Sent: Monday, August 23, 2010 2:01 AM
>>>> To: Joel Ebrahimi
>>>> Cc: Will Metcalf; oisf-devel at openinfosecfoundation.org
>>>> Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>> Hi Joel,
>>> For the pcap module we pass a variable to pcap_dispatch that makes
>>> we read a max number of packets equal to or lower than 256, based on
>>>> the
>>> amount of available packets in the PacketPool. If none are available
>>> wait. See lines 180 to 200 in source-pcap.c
>>> Cheers,
>>> Victor
>>> Joel Ebrahimi wrote:
>>>>>> Sure. As I said though its not always a segfault but I can see
>>>> the
>>>>>> memory being used up on the system. Here is the segfault
>>>>>> #0  0x10012150 in PacketEnqueue (q=0x1080eb70, p=0x2a9) at
>>>>>> packet-queue.c:142
>>>>>> #1  0x10012f98 in ReceivePcap (tv=0x1080ea78, p=0x10265f70,
>>>>>> data=0x107faef8, pq=0x1080eb10, postpq=0x1080eb70)
>>>>>>     at source-pcap.c:219
>>>>>> #2  0x10104f44 in TmThreadsSlot1 (td=0x1080ea78) at
>>>>>> #3  0x0ff0d994 in start_thread () from /lib/libpthread.so.0
>>>>>> #4  0x0ff0d994 in start_thread () from /lib/libpthread.so.0
>>>>>> Previous frame inner to this frame (corrupt stack?)
>>>>>> Really the problem seems to get ugly in the ReceivePcap function
>>>> here:
>>>>>>     for (cnt = 0; cnt < ptv->array_idx; cnt++) {
>>>>>>         Packet *pp = ptv->array[cnt];
>>>>>>         /* enqueue all but the first in the postpq, the first
>>>>>>          * pkt is handled by the tv "out handler" */
>>>>>>         if (cnt > 0) {
>>>>>>             PacketEnqueue(postpq, pp);
>>>>>>         }
>>>>>>     }
>>>>>> The problem being that array_idx keeps incrementing. So in that
>>>> specific
>>>>>> backtrace case above array_idx is 6538 and when this section of
>>>> is
>>>>>> hit, once count exceeds 255 we are outside the range of memory
>>>>>> you can see from the backtrace that in the function PacketEnqueue
>>>> where
>>>>>> p's address is wrong.
>>>>>> I know the array stuff is new in the 1.0.x release and looking it
>>>> over I
>>>>>> don't see a case that ensures it stays within the max 256
>>>> or I
>>>>>> don't see a case where array_idx ever gets reset.
>>>>>> // Joel
>>>>>>> -----Original Message-----
>>>>>>> From: Will Metcalf [mailto:william.metcalf at gmail.com]
>>>>>>> Sent: Thursday, August 19, 2010 4:36 PM
>>>>>>> To: Joel Ebrahimi
>>>>>>> Cc: oisf-devel at openinfosecfoundation.org
>>>>>>> Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>>>>>> Can you give us the output of a backtrace Joel?  You can send
>>>> off-list
>>>>>>> if you so desire.
>>>>>>> Regards,
>>>>>>> Will
>>>>>>> On Thu, Aug 19, 2010 at 5:19 PM, Joel Ebrahimi
><jebrahimi at bivio.net>
>>>>>> wrote:
>>>>>>>> I'm seeing multiple problems with the array that was added to
>>>>>>>> PcapThreadVars_.
>>>>>>>> I either exhaust memory completely or segfault. I think they
>>>>>> stem
>>>>>>>> from the same issue with is that the array seems to keep
>>>> and
>>>>>>>> eventually there is an access outside of PCAP_FILE_MAX_PKTS
>>>>>> which
>>>>>>>> causes the segfaults.
>>>>>>>> Is the array supposed to hold a finite number of past packets
>>>> 256
>>>>>> or
>>>>>>>> whatever PCAP_FILE_MAX_PKTS is set to? If so then I suppose a
>>>> modulus
>>>>>>>> operator or some kind of reset is needed.
>>>>>>>> // Joel
>>>>>>>> Joel Ebrahimi
>>>>>>>> Solutions Architect
>>>>>>>> Bivio Networks
>>>>>>>> jebrahimi at bivio.net
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-devel mailing list
>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>> _______________________________________________
>>>>>> Oisf-devel mailing list
>>>>>> Oisf-devel at openinfosecfoundation.org
>>- --
>>- ---------------------------------------------
>>Victor Julien
>>PGP: http://www.inliniac.net/victorjulien.asc
>>- ---------------------------------------------
>>Version: GnuPG v1.4.9 (GNU/Linux)
>>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>Oisf-devel mailing list
>Oisf-devel at openinfosecfoundation.org

More information about the Oisf-devel mailing list