[Oisf-users] TCP reassembly gaps

[Victor Julien]

On 04/18/2012 08:26 PM, Chris Wakelin wrote:
> I've been noticing something a bit odd on our student network sensor:
>> -------------------------------------------------------------------
>> Date: 4/18/2012 -- 19:17:24 (uptime: 0d, 00h 27m 05s)
>> -------------------------------------------------------------------
>> Counter                   | TM Name                   | Value
>> -------------------------------------------------------------------
>> flow_mgr.closed_pruned    | FlowManagerThread         | 437460
>> flow_mgr.new_pruned       | FlowManagerThread         | 428581
>> flow_mgr.est_pruned       | FlowManagerThread         | 231037
>> flow.memuse               | FlowManagerThread         | 49419632
>> flow.spare                | FlowManagerThread         | 40690
>> flow.emerg_mode_entered   | FlowManagerThread         | 0
>> flow.emerg_mode_over      | FlowManagerThread         | 0
> ...
>> decoder.pkts              | RxPFReth16                | 16005410
>> decoder.bytes             | RxPFReth16                | 10858104740
>> decoder.ipv4              | RxPFReth16                | 16021667
>> decoder.ipv6              | RxPFReth16                | 1292
>> decoder.ethernet          | RxPFReth16                | 16005410
>> decoder.raw               | RxPFReth16                | 0
>> decoder.sll               | RxPFReth16                | 0
>> decoder.tcp               | RxPFReth16                | 9280996
>> decoder.udp               | RxPFReth16                | 5480993
>> decoder.sctp              | RxPFReth16                | 0
>> decoder.icmpv4            | RxPFReth16                | 8812
>> decoder.icmpv6            | RxPFReth16                | 1082
>> decoder.ppp               | RxPFReth16                | 13353
>> decoder.pppoe             | RxPFReth16                | 0
>> decoder.gre               | RxPFReth16                | 13354
>> decoder.vlan              | RxPFReth16                | 7257415
>> decoder.avg_pkt_size      | RxPFReth16                | 678
>> decoder.max_pkt_size      | RxPFReth16                | 1522
>> defrag.ipv4.fragments     | RxPFReth16                | 33338
>> defrag.ipv4.reassembled   | RxPFReth16                | 16342
>> defrag.ipv4.timeouts      | RxPFReth16                | 0
>> defrag.ipv6.fragments     | RxPFReth16                | 0
>> defrag.ipv6.reassembled   | RxPFReth16                | 0
>> defrag.ipv6.timeouts      | RxPFReth16                | 0
>> tcp.sessions              | RxPFReth16                | 72635
>> tcp.ssn_memcap_drop       | RxPFReth16                | 0
>> tcp.pseudo                | RxPFReth16                | 9940
>> tcp.invalid_checksum      | RxPFReth16                | 0
>> tcp.no_flow               | RxPFReth16                | 0
>> tcp.reused_ssn            | RxPFReth16                | 0
>> tcp.memuse                | RxPFReth16                | 6029312
>> tcp.syn                   | RxPFReth16                | 115837
>> tcp.synack                | RxPFReth16                | 70546
>> tcp.rst                   | RxPFReth16                | 35533
>> tcp.segment_memcap_drop   | RxPFReth16                | 0
>> tcp.stream_depth_reached  | RxPFReth16                | 160
>> tcp.reassembly_memuse     | RxPFReth16                | 1641134685
>> tcp.reassembly_gap        | RxPFReth16                | 10002
>> detect.alert              | RxPFReth16                | 310
> 
> The tcp.reassembly_gap count (added over all 6 threads) is increasing at
> about 40/sec.
> 
> This is using an Intel 10GB (ixgbe) card, and PF_RING reckons there are
> no lost packets. The network load is about 300-400Mb/s rising to nearly
> 1GB when the students are all here. (One oddity about this port mirror
> is that the packets are VLAN-tagged in only one direction. Extreme
> Networks say this is by design :-$; I've modified the PF_RING packet
> hash to ignore VLAN tags)
> 
> On the main campus network, 1GB port mirror (VLAN-tagged properly) there
> are no gaps, even though it's frequently losing packets (e.g. when the
> traffic goes over 1GB).

> Any idea how do debug this? Could it be an ethernet driver issue?

This is strange indeed.

One way to debug is to enable this rule:

# Sequence gap: missing data in the reassembly engine. Usually due to
packet loss. Will be very noisy on a overloaded link / sensor.
alert tcp any any -> any any (msg:"SURICATA STREAM reassembly sequence
GAP -- missing packet(s)"; stream-event:reassembly_seq_gap; sid:2210048;
rev:1;)

You may want to threshold it some.

Then look at the streams that fire...

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------