[Oisf-users] TCP reassembly gaps

[Victor Julien]

On 04/19/2012 11:33 AM, Chris Wakelin wrote:
> On 19/04/12 06:33, Victor Julien wrote:
>>> The tcp.reassembly_gap count (added over all 6 threads) is increasing at
>>> about 40/sec.
>>>
>>> This is using an Intel 10GB (ixgbe) card, and PF_RING reckons there are
>>> no lost packets. The network load is about 300-400Mb/s rising to nearly
>>> 1GB when the students are all here. (One oddity about this port mirror
>>> is that the packets are VLAN-tagged in only one direction. Extreme
>>> Networks say this is by design :-$; I've modified the PF_RING packet
>>> hash to ignore VLAN tags)
>>>
>>> On the main campus network, 1GB port mirror (VLAN-tagged properly) there
>>> are no gaps, even though it's frequently losing packets (e.g. when the
>>> traffic goes over 1GB).
>>
>>> Any idea how do debug this? Could it be an ethernet driver issue?
>>
>> This is strange indeed.
>>
>> One way to debug is to enable this rule:
>>
>> # Sequence gap: missing data in the reassembly engine. Usually due to
>> packet loss. Will be very noisy on a overloaded link / sensor.
>> alert tcp any any -> any any (msg:"SURICATA STREAM reassembly sequence
>> GAP -- missing packet(s)"; stream-event:reassembly_seq_gap; sid:2210048;
>> rev:1;)
> 
> Ah, I forgot we could sig these now!
> 
>>
>> You may want to threshold it some.
>>
>> Then look at the streams that fire...
>>
> 
> I think it must be a PF_RING/ixgbe issue. I've got IRQ-pinning on and
> RSS enabled so it might be worth trying with RSS turned off.
> 
> I created a pcap (~200MB/250K packets) with PF-RING-enabled tcpdump for
> a minute or so, filtering on a Google/Youtube /24 network with "net
> 64.156.119.0/24 or (vlan and net 64.156.119.0/24)" to get varied sources
> and destinations, and Wireshark agrees it's missing packets. PF_RING
> stats suggest no dropped packets though.
> 
> The above sig hits 45 times and gave me some src/dst pairs to check in
> Wireshark :)

So you're saying you're loosing packets after all? I guess that could
make sense. PF_RING can't account for packets that are lost before
reaching the NIC. The gap counter is an indication of packet loss.


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------