[Oisf-users] Packet Loss

[Peter Manev]

On Mon, Jun 9, 2014 at 5:44 PM, Yasha Zislin <coolyasha at hotmail.com> wrote:
> I've done some additional testing.
>
> I've ran pfcount with 16 threads with the same parameters as Suricata does.
> I've had only one instance of /proc/net/pf_ring instantiated but 16 threads
> in processes (TOP -H).
>
> I've been running it for an hour with 0 packet loss. PF_RING slot usage does
> not go above 200 (with 688k total).
>
> So my packet loss occurs due to Suricata and not network/pf_ring related.
>
> Thanks.


What is the output of:

ethtool -g eth15
and
ethtool -g eth17


>
>> Date: Mon, 9 Jun 2014 16:30:25 +0200
>
>> Subject: Re: [Oisf-users] Packet Loss
>> From: petermanev at gmail.com
>> To: coolyasha at hotmail.com; oisf-users at lists.openinfosecfoundation.org
>
>>
>> On Mon, Jun 9, 2014 at 4:21 PM, Yasha Zislin <coolyasha at hotmail.com>
>> wrote:
>> > So these two interfaces are kind of related. If traffic spikes on one,
>> > it
>> > will increase on the other.
>> > In fact, these two span ports see traffic before and after the firewall.
>> > So
>> > traffic is the same except for NATting piece. After the firewall IPs
>> > change
>> > to private addressing.
>> > Not sure if this scenario can have issues, BUT my packet drop was
>> > occurring
>> > when I was just trying to get Suricata to run with one interface.
>> > Important point is that CPUs dont hit 100% (it was doing that in the
>> > beginning until I started to offload everything to RAM).
>> >
>> > You are correct about memory consumption of threads. I've increased
>> > min_num_slots by running this:
>> > modprobe pf_ring transparent_mode=0 min_num_slots=400000
>> > enable_tx_capture=0
>> >
>> > cat /proc/net/pf_ring/info
>> > PF_RING Version : 6.0.2 ($Revision: exported$)
>> > Total rings : 16
>> >
>> > Standard (non DNA) Options
>> > Ring slots : 400000
>> > Slot version : 15
>> > Capture TX : No [RX only]
>> > IP Defragment : No
>> > Socket Mode : Standard
>> > Transparent mode : Yes [mode 0]
>> > Total plugins : 0
>> > Cluster Fragment Queue : 3852
>> > Cluster Fragment Discard : 428212
>> >
>> > I could have increased it higher (it let me do that) but Free Num Slots
>> > stopped increased after 400000 value.
>> > I did notice if I set min_num_slots to default 65k number, Free Num
>> > Slots
>> > get to 0 faster and packet drop begins.
>> >
>> > Here is a stat for one of the threads:
>> > cat /proc/net/pf_ring/6224-eth17.595
>> > Bound Device(s) : eth17
>> > Active : 1
>> > Breed : Non-DNA
>> > Sampling Rate : 1
>> > Capture Direction : RX+TX
>> > Socket Mode : RX+TX
>> > Appl. Name : Suricata
>> > IP Defragment : No
>> > BPF Filtering : Disabled
>> > # Sw Filt. Rules : 0
>> > # Hw Filt. Rules : 0
>> > Poll Pkt Watermark : 128
>> > Num Poll Calls : 6408432
>> > Channel Id Mask : 0xFFFFFFFF
>> > Cluster Id : 99
>> > Slot Version : 15 [6.0.2]
>> > Min Num Slots : 688290
>> > Bucket Len : 1522
>> > Slot Len : 1560 [bucket+header]
>> > Tot Memory : 1073741824
>> > Tot Packets : 902405618
>> > Tot Pkt Lost : 79757335
>> > Tot Insert : 822648289
>> > Tot Read : 822648236
>> > Insert Offset : 219035272
>> > Remove Offset : 218997656
>> > TX: Send Ok : 0
>> > TX: Send Errors : 0
>> > Reflect: Fwd Ok : 0
>> > Reflect: Fwd Errors: 0
>> > Num Free Slots : 688237
>> >
>> >
>> > For NICs I have 10 gig Fiber HP nic (I think with Qlogic chip).
>> >
>> > BTW, I had to configure both PF_RING interfaces to be the same cluster
>> > ID.
>> > For some reason, setting them to different numbers would not work.
>> >
>> > You are correct about the behavior. It runs fine, Free Num Slots for ALL
>> > threads get to 0, packet drop starts, after some time Free Num Slots go
>> > back
>> > to almost 100% available and packet drop stops.
>> > It feels like it is getting choked on something and starts to fill up.
>> >
>> > Timeout values are as follows:
>> > flow-timeouts:
>> >
>> > default:
>> > new: 3
>> > established: 30
>> > closed: 0
>> > emergency-new: 10
>> > emergency-established: 10
>> > emergency-closed: 0
>> > tcp:
>> > new: 6
>> > established: 100
>> > closed: 12
>> > emergency-new: 1
>> > emergency-established: 5
>> > emergency-closed: 2
>> > udp:
>> > new: 3
>> > established: 30
>> > emergency-new: 3
>> > emergency-established: 10
>> > icmp:
>> > new: 3
>> > established: 30
>> > emergency-new: 1
>> > emergency-established: 10
>> >
>> > I set them just like one of these "10gig and beyond" articles said.
>> >
>> > Thank you for your help.
>> >
>>
>> Thank you for the feedback.
>> Please keep the conversation on the list :)
>>
>>
>> --
>> Regards,
>> Peter Manev



-- 
Regards,
Peter Manev