[Oisf-devel] Packet Array in 1.0.x
Joel Ebrahimi
jebrahimi at bivio.net
Tue Sep 7 20:34:13 UTC 2010
Victor,
I have looked over the code and I don't see any checks that help. Let me
explain the 2 problems I see.
First when pcap_dispatch is called I have a large number of packets that
are available. So this in turn calls PcapCallback. Now something
PcapCallback is called 15,000 times in a row which will instantiate a
lot of memory with the calls to PacketGetFromQueueOrAlloc(), so in this
case I exhaust the entire memory of the system creating buffers for
15,000 packets. So this results in Linux killing the process.
In the other case I do not exhaust all the memory, lets say PcapCallback
is only called 12,000 times. Now in this case I return from the callback
and move into the for loop. So at this point array_idx is 12,000. So
when I enter the for loop cnt is then incremented up to 12,000. The
problem here though is that the ptv->array only has 256 slots allocated
for it, so since cnt exceeds 256 and there is an access to the variable
ptv->array[cnt]; this will cause a segfault.
Does that explain whats happening a little clearer? Im not sure why no
one else has seen this yet, unless their buffers are small and the
PcapCallback routine is called less than 256 times before processing.
// Joel
>-----Original Message-----
>From: oisf-devel-bounces at openinfosecfoundation.org [mailto:oisf-devel-
>bounces at openinfosecfoundation.org] On Behalf Of Joel Ebrahimi
>Sent: Tuesday, September 07, 2010 10:01 AM
>To: Victor Julien
>Cc: oisf-devel at openinfosecfoundation.org
>Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>
>Victor,
>
>Just to be clear, this problem is occurring in the Suricata code base
>and not within the Bivio modifications. The code Im testing this with
is
>unmodified.
>
>// Joel
>
>>-----Original Message-----
>>From: Victor Julien [mailto:victor at inliniac.net]
>>Sent: Monday, September 06, 2010 7:55 AM
>>To: Joel Ebrahimi
>>Cc: oisf-devel at openinfosecfoundation.org; Will Metcalf
>>Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA1
>>
>>Joel Ebrahimi wrote:
>>> Hey Victor,
>>>
>>> In the first call to pcap_dispatch, when this statement:
>>>
>>> (pcap_max_read_packets < packet_q_len) ? *pcap_max_read_packets
>:
>>> packet_q_len
>>>
>>> pcap_max_read_packet is 0, so 0 is passed in to cnt for
>pcap_dispatch. A
>>> 0 or a -1 here causes the all the packets received in the buffer to
>be
>>> processed.
>>
>>Right. In source-pcap.c we make sure we never get to that condition by
>>entering a wait loop that sets packet_q_len based on the number of
>>available packets in the packet pool. As long as it's 0 it will remain
>>in that loop.
>>
>>Did you implement a similar protection?
>>
>>> In my test cast I can see PcapCallback called 10's or thousands or
>>> times. It appear that depending on the pcap file Im sending that
>either
>>> memory is exhausted here in PcapCallback and the process is killed
or
>if
>>> it returns then the variable ptv->array_idx is set to something
>>> extremely high (like 15000) and when the for loop happens after the
>>> pcap_dispatch, then once cnt goes passed 256 we are accessing
>outside
>>> our memory space and I get the segfault.
>>
>>Problem is that it will be hard to set the array size to be big
enough.
>> So I think choosing and then enforcing a limit is inevitable.
>>
>>Cheers,
>>Victor
>>
>>>
>>> // Joel
>>>
>>>> -----Original Message-----
>>>> From: Victor Julien [mailto:victor at inliniac.net]
>>>> Sent: Monday, August 23, 2010 2:01 AM
>>>> To: Joel Ebrahimi
>>>> Cc: Will Metcalf; oisf-devel at openinfosecfoundation.org
>>>> Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>>>
>>> Hi Joel,
>>>
>>> For the pcap module we pass a variable to pcap_dispatch that makes
>sure
>>> we read a max number of packets equal to or lower than 256, based on
>>>> the
>>> amount of available packets in the PacketPool. If none are available
>we
>>> wait. See lines 180 to 200 in source-pcap.c
>>>
>>> Cheers,
>>> Victor
>>>
>>> Joel Ebrahimi wrote:
>>>>>> Sure. As I said though its not always a segfault but I can see
all
>>>> the
>>>>>> memory being used up on the system. Here is the segfault
>backtrace:
>>>>>>
>>>>>> #0 0x10012150 in PacketEnqueue (q=0x1080eb70, p=0x2a9) at
>>>>>> packet-queue.c:142
>>>>>> #1 0x10012f98 in ReceivePcap (tv=0x1080ea78, p=0x10265f70,
>>>>>> data=0x107faef8, pq=0x1080eb10, postpq=0x1080eb70)
>>>>>> at source-pcap.c:219
>>>>>> #2 0x10104f44 in TmThreadsSlot1 (td=0x1080ea78) at
>tm-threads.c:371
>>>>>> #3 0x0ff0d994 in start_thread () from /lib/libpthread.so.0
>>>>>> #4 0x0ff0d994 in start_thread () from /lib/libpthread.so.0
>>>>>> Previous frame inner to this frame (corrupt stack?)
>>>>>>
>>>>>> Really the problem seems to get ugly in the ReceivePcap function
>>>> here:
>>>>>> for (cnt = 0; cnt < ptv->array_idx; cnt++) {
>>>>>> Packet *pp = ptv->array[cnt];
>>>>>>
>>>>>> /* enqueue all but the first in the postpq, the first
>>>>>> * pkt is handled by the tv "out handler" */
>>>>>> if (cnt > 0) {
>>>>>> PacketEnqueue(postpq, pp);
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> The problem being that array_idx keeps incrementing. So in that
>>>> specific
>>>>>> backtrace case above array_idx is 6538 and when this section of
>code
>>>> is
>>>>>> hit, once count exceeds 255 we are outside the range of memory
and
>as
>>>>>> you can see from the backtrace that in the function PacketEnqueue
>>>> where
>>>>>> p's address is wrong.
>>>>>>
>>>>>> I know the array stuff is new in the 1.0.x release and looking it
>>>> over I
>>>>>> don't see a case that ensures it stays within the max 256
>allocated
>>>> or I
>>>>>> don't see a case where array_idx ever gets reset.
>>>>>>
>>>>>> // Joel
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Will Metcalf [mailto:william.metcalf at gmail.com]
>>>>>>> Sent: Thursday, August 19, 2010 4:36 PM
>>>>>>> To: Joel Ebrahimi
>>>>>>> Cc: oisf-devel at openinfosecfoundation.org
>>>>>>> Subject: Re: [Oisf-devel] Packet Array in 1.0.x
>>>>>>>
>>>>>>> Can you give us the output of a backtrace Joel? You can send
>>>> off-list
>>>>>>> if you so desire.
>>>>>>>
>>>>>>> Regards,
>>>>>>>
>>>>>>> Will
>>>>>>>
>>>>>>> On Thu, Aug 19, 2010 at 5:19 PM, Joel Ebrahimi
><jebrahimi at bivio.net>
>>>>>> wrote:
>>>>>>>> I'm seeing multiple problems with the array that was added to
>>>>>>>> PcapThreadVars_.
>>>>>>>>
>>>>>>>> I either exhaust memory completely or segfault. I think they
>both
>>>>>> stem
>>>>>>>> from the same issue with is that the array seems to keep
filling
>>>> and
>>>>>>>> eventually there is an access outside of PCAP_FILE_MAX_PKTS
>(256)
>>>>>> which
>>>>>>>> causes the segfaults.
>>>>>>>>
>>>>>>>> Is the array supposed to hold a finite number of past packets
>ie.
>>>> 256
>>>>>> or
>>>>>>>> whatever PCAP_FILE_MAX_PKTS is set to? If so then I suppose a
>>>> modulus
>>>>>>>> operator or some kind of reset is needed.
>>>>>>>>
>>>>>>>> // Joel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Joel Ebrahimi
>>>>>>>> Solutions Architect
>>>>>>>> Bivio Networks
>>>>>>>> jebrahimi at bivio.net
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-devel mailing list
>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>>
>http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>
>>>>>> _______________________________________________
>>>>>> Oisf-devel mailing list
>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>
>>
>>- --
>>- ---------------------------------------------
>>Victor Julien
>>http://www.inliniac.net/
>>PGP: http://www.inliniac.net/victorjulien.asc
>>- ---------------------------------------------
>>
>>-----BEGIN PGP SIGNATURE-----
>>Version: GnuPG v1.4.9 (GNU/Linux)
>>Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>>
>>iEYEARECAAYFAkyFAMkACgkQiSMBBAuniMd6iQCeMytxiSc1DigDWNQd9qH+BVC0
>>GMwAn3TfL3rN8uvEGJZcXo3fU2kY0wk+
>>=dBGn
>>-----END PGP SIGNATURE-----
>_______________________________________________
>Oisf-devel mailing list
>Oisf-devel at openinfosecfoundation.org
>http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
More information about the Oisf-devel
mailing list