[Oisf-users] TCP reassembly gaps

[Chris Wakelin]

I've been noticing something a bit odd on our student network sensor:
> -------------------------------------------------------------------
> Date: 4/18/2012 -- 19:17:24 (uptime: 0d, 00h 27m 05s)
> -------------------------------------------------------------------
> Counter                   | TM Name                   | Value
> -------------------------------------------------------------------
> flow_mgr.closed_pruned    | FlowManagerThread         | 437460
> flow_mgr.new_pruned       | FlowManagerThread         | 428581
> flow_mgr.est_pruned       | FlowManagerThread         | 231037
> flow.memuse               | FlowManagerThread         | 49419632
> flow.spare                | FlowManagerThread         | 40690
> flow.emerg_mode_entered   | FlowManagerThread         | 0
> flow.emerg_mode_over      | FlowManagerThread         | 0
...
> decoder.pkts              | RxPFReth16                | 16005410
> decoder.bytes             | RxPFReth16                | 10858104740
> decoder.ipv4              | RxPFReth16                | 16021667
> decoder.ipv6              | RxPFReth16                | 1292
> decoder.ethernet          | RxPFReth16                | 16005410
> decoder.raw               | RxPFReth16                | 0
> decoder.sll               | RxPFReth16                | 0
> decoder.tcp               | RxPFReth16                | 9280996
> decoder.udp               | RxPFReth16                | 5480993
> decoder.sctp              | RxPFReth16                | 0
> decoder.icmpv4            | RxPFReth16                | 8812
> decoder.icmpv6            | RxPFReth16                | 1082
> decoder.ppp               | RxPFReth16                | 13353
> decoder.pppoe             | RxPFReth16                | 0
> decoder.gre               | RxPFReth16                | 13354
> decoder.vlan              | RxPFReth16                | 7257415
> decoder.avg_pkt_size      | RxPFReth16                | 678
> decoder.max_pkt_size      | RxPFReth16                | 1522
> defrag.ipv4.fragments     | RxPFReth16                | 33338
> defrag.ipv4.reassembled   | RxPFReth16                | 16342
> defrag.ipv4.timeouts      | RxPFReth16                | 0
> defrag.ipv6.fragments     | RxPFReth16                | 0
> defrag.ipv6.reassembled   | RxPFReth16                | 0
> defrag.ipv6.timeouts      | RxPFReth16                | 0
> tcp.sessions              | RxPFReth16                | 72635
> tcp.ssn_memcap_drop       | RxPFReth16                | 0
> tcp.pseudo                | RxPFReth16                | 9940
> tcp.invalid_checksum      | RxPFReth16                | 0
> tcp.no_flow               | RxPFReth16                | 0
> tcp.reused_ssn            | RxPFReth16                | 0
> tcp.memuse                | RxPFReth16                | 6029312
> tcp.syn                   | RxPFReth16                | 115837
> tcp.synack                | RxPFReth16                | 70546
> tcp.rst                   | RxPFReth16                | 35533
> tcp.segment_memcap_drop   | RxPFReth16                | 0
> tcp.stream_depth_reached  | RxPFReth16                | 160
> tcp.reassembly_memuse     | RxPFReth16                | 1641134685
> tcp.reassembly_gap        | RxPFReth16                | 10002
> detect.alert              | RxPFReth16                | 310

The tcp.reassembly_gap count (added over all 6 threads) is increasing at
about 40/sec.

This is using an Intel 10GB (ixgbe) card, and PF_RING reckons there are
no lost packets. The network load is about 300-400Mb/s rising to nearly
1GB when the students are all here. (One oddity about this port mirror
is that the packets are VLAN-tagged in only one direction. Extreme
Networks say this is by design :-$; I've modified the PF_RING packet
hash to ignore VLAN tags)

On the main campus network, 1GB port mirror (VLAN-tagged properly) there
are no gaps, even though it's frequently losing packets (e.g. when the
traffic goes over 1GB).

Any idea how do debug this? Could it be an ethernet driver issue?

Best Wishes,
Chris

-- 
--+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+---+-
Christopher Wakelin,                           c.d.wakelin at reading.ac.uk
IT Services Centre, The University of Reading,  Tel: +44 (0)118 378 2908
Whiteknights, Reading, RG6 6AF, UK              Fax: +44 (0)118 975 3094