[Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Peter Manev]

On Thu, Feb 21, 2019 at 5:19 PM Nelson, Cooper <cnelson at ucsd.edu> wrote:
>
> No guarantee this is what you are seeing, but I had the exact same issue and it was due to our networking folks deploying some of these things on our network:
>
> https://www.perfsonar.net/
>
> They were generating large amounts of jumbo frames up to 64k, which were causing packet drops in the millions when they do a performance test.  Filtering them on our Arista solved the problem.

Could be related indeed.

@Sean Could you try the following and give me some feedback please.

Could you compile and install  ethtool for your specific kernel (the
example below assumes 4.19 for example, substitute it with your kernel
version please):


wget https://mirrors.edge.kernel.org/pub/software/network/ethtool/ethtool-4.19.tar.xz
&& \
tar -xf ethtool-4.19.tar.xz && \
cd ethtool-4.19 && \
./configure && make clean && make && make install

It should end up in
ls -lh /usr/local/sbin/ethtool

Then could you please  share the output of
/usr/local/sbin/ethtool --show-rxfh  interface_name_here
and the NIC model?

Thank you


>
> -Coop
>
> -----Original Message-----
> From: Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org> On Behalf Of Cloherty, Sean E
> Sent: Thursday, February 21, 2019 7:33 AM
> To: Peter Manev <petermanev at gmail.com>
> Cc: Open Information Security Foundation <oisf-users at lists.openinfosecfoundation.org>
> Subject: Re: [Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support
>
> Hello Peter -
>
> I started 4.1.2 yesterdat at 15:41 local time -which mean that I missed the big mid-day volume of traffic.  However it still dropped almost 18 million packets.  Not over time but in a short burst around 21:00.  That behavior is something  I have observed  since at least as far back as 2016.  When Suricata drops packet on any of my tuned systems, it happens in short bursts from one entry in the stats log to the next (5 minutes) and then stop.  More info:
>
> *These bursts seem to last no longer than 5-10 minutes and then are stable for hours /days / weeks.
>
> * The numbers dropped. are usually in the millions. In testing on 4.1.2 it went from no drops at startup at 15:41 and then between 22:12 and 22:16 it dropped 17.92 million packets.
>
> * Traffic volume doesn't seem to correlate to packet drops.  At the time I fired the 4.1.2. host up, the avg volume was 1.72Gbps with peaks close to 3.1 Gbps.  At the time when the 17 million packets dropped, the avg volume was 640 Gbps with peaks around 1.35 Gbps.
>
> In the past I would see a normal linear increase in packet loss over time.  Once you helped with tuning, that almost never happened.  SEPTun reduced it further.  I monitor CPU Use / RAM Use / Interrupts / and Suricata stats via Zabbix so I can do a pretty quick comparison of when the drops happen and what else is going on at the time so I am at a bit of a loss where there seems to be no correlation.
>
> Sean.
>
>


-- 
Regards,
Peter Manev