[Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

Fri Feb 22 18:16:11 UTC 2019

Below is the info you aske for and a bit more....and I am looking into whether or not our networking group is using any tools similar to that which Cooper mentioned.  Odd traffic generators are always a good thing to look for but not usually my 1st thought.

ETHTOOL:

[root at idstest ethtool-4.19]# /usr/local/sbin/ethtool --show-rxfh ens1f1
RX flow hash indirection table for ens1f1 with 1 RX ring(s):
    0:      0     0     0     0     0     0     0     0
    8:      0     0     0     0     0     0     0     0
   16:      0     0     0     0     0     0     0     0
   24:      0     0     0     0     0     0     0     0
   32:      0     0     0     0     0     0     0     0
   40:      0     0     0     0     0     0     0     0
   48:      0     0     0     0     0     0     0     0
   56:      0     0     0     0     0     0     0     0
   64:      0     0     0     0     0     0     0     0
   72:      0     0     0     0     0     0     0     0
   80:      0     0     0     0     0     0     0     0
   88:      0     0     0     0     0     0     0     0
   96:      0     0     0     0     0     0     0     0
  104:      0     0     0     0     0     0     0     0
  112:      0     0     0     0     0     0     0     0
  120:      0     0     0     0     0     0     0     0
RSS hash key:
a0:ca:58:fd:12:2a:68:2a:cd:f3:07:3b:be:2b:47:3e:14:b3:23:94:7f:dc:15:ec:38:8a:12:b1:4b:6f:b0:0c:87:fb:fe:9f:76:6e:7a:9c
RSS hash function:
    toeplitz: on
    xor: off
    crc32: off

NIC INFO:

81:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
81:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)

These commands run in the suricata startup script :

killall irqbalance

/usr/local/src/ixgbe-5.5.2/scripts/set_irq_affinity 9 ens1f1

ethtool -A ens1f1 rx off tx off
ethtool -K ens1f1 sg off
ethtool -K ens1f1 gro off
ethtool -K ens1f1 lro off
ethtool -K ens1f1 tso off
ethtool -K ens1f1 gso off
ethtool -K ens1f1 rx off
ethtool -K ens1f1 tx off
ethtool -K ens1f1 rxvlan off
ethtool -K ens1f1 txvlan off
ethtool --offload ens1f1 rx off tx off
ethtool -G ens1f1 rx 512
ethtool -L ens1f1 combined 1

LD_PRELOAD="/usr/lib64/libtcmalloc_minimal.so.4" /usr/bin/suricata -c /etc/suricata/suricata.yaml -v --af-packet=ens1f1 --runmode=workers -D

-----Original Message-----
From: Peter Manev <petermanev at gmail.com> 
Sent: Friday, February 22, 2019 12:43 PM
To: Nelson, Cooper <cnelson at ucsd.edu>
Cc: Cloherty, Sean E <scloherty at mitre.org>; Open Information Security Foundation <oisf-users at lists.openinfosecfoundation.org>
Subject: Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

On Thu, Feb 21, 2019 at 5:19 PM Nelson, Cooper <cnelson at ucsd.edu> wrote:
>
> No guarantee this is what you are seeing, but I had the exact same issue and it was due to our networking folks deploying some of these things on our network:
>
> https://www.perfsonar.net/
>
> They were generating large amounts of jumbo frames up to 64k, which were causing packet drops in the millions when they do a performance test.  Filtering them on our Arista solved the problem.

Could be related indeed.

@Sean Could you try the following and give me some feedback please.

Could you compile and install  ethtool for your specific kernel (the example below assumes 4.19 for example, substitute it with your kernel version please):

wget https://mirrors.edge.kernel.org/pub/software/network/ethtool/ethtool-4.19.tar.xz
&& \
tar -xf ethtool-4.19.tar.xz && \
cd ethtool-4.19 && \
./configure && make clean && make && make install

It should end up in
ls -lh /usr/local/sbin/ethtool

Then could you please  share the output of /usr/local/sbin/ethtool --show-rxfh  interface_name_here and the NIC model?

Thank you

>
> -Coop
>
> -----Original Message-----
> From: Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org> 
> On Behalf Of Cloherty, Sean E
> Sent: Thursday, February 21, 2019 7:33 AM
> To: Peter Manev <petermanev at gmail.com>
> Cc: Open Information Security Foundation 
> <oisf-users at lists.openinfosecfoundation.org>
> Subject: Re: [Oisf-users] [EXT] Re: Packet loss and increased resource 
> consumption after upgrade to 4.1.2 with Rust support
>
> Hello Peter -
>
> I started 4.1.2 yesterdat at 15:41 local time -which mean that I missed the big mid-day volume of traffic.  However it still dropped almost 18 million packets.  Not over time but in a short burst around 21:00.  That behavior is something  I have observed  since at least as far back as 2016.  When Suricata drops packet on any of my tuned systems, it happens in short bursts from one entry in the stats log to the next (5 minutes) and then stop.  More info:
>
> *These bursts seem to last no longer than 5-10 minutes and then are stable for hours /days / weeks.
>
> * The numbers dropped. are usually in the millions. In testing on 4.1.2 it went from no drops at startup at 15:41 and then between 22:12 and 22:16 it dropped 17.92 million packets.
>
> * Traffic volume doesn't seem to correlate to packet drops.  At the time I fired the 4.1.2. host up, the avg volume was 1.72Gbps with peaks close to 3.1 Gbps.  At the time when the 17 million packets dropped, the avg volume was 640 Gbps with peaks around 1.35 Gbps.
>
> In the past I would see a normal linear increase in packet loss over time.  Once you helped with tuning, that almost never happened.  SEPTun reduced it further.  I monitor CPU Use / RAM Use / Interrupts / and Suricata stats via Zabbix so I can do a pretty quick comparison of when the drops happen and what else is going on at the time so I am at a bit of a loss where there seems to be no correlation.
>
> Sean.
>
>

--
Regards,
Peter Manev