[Oisf-users] [EXT] Re: Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

Thu Feb 21 15:32:31 UTC 2019

Hello Peter -

I started 4.1.2 yesterdat at 15:41 local time -which mean that I missed the big mid-day volume of traffic.  However it still dropped almost 18 million packets.  Not over time but in a short burst around 21:00.  That behavior is something  I have observed  since at least as far back as 2016.  When Suricata drops packet on any of my tuned systems, it happens in short bursts from one entry in the stats log to the next (5 minutes) and then stop.  More info:

*These bursts seem to last no longer than 5-10 minutes and then are stable for hours /days / weeks.

* The numbers dropped. are usually in the millions. In testing on 4.1.2 it went from no drops at startup at 15:41 and then between 22:12 and 22:16 it dropped 17.92 million packets. 

* Traffic volume doesn't seem to correlate to packet drops.  At the time I fired the 4.1.2. host up, the avg volume was 1.72Gbps with peaks close to 3.1 Gbps.  At the time when the 17 million packets dropped, the avg volume was 640 Gbps with peaks around 1.35 Gbps.

In the past I would see a normal linear increase in packet loss over time.  Once you helped with tuning, that almost never happened.  SEPTun reduced it further.  I monitor CPU Use / RAM Use / Interrupts / and Suricata stats via Zabbix so I can do a pretty quick comparison of when the drops happen and what else is going on at the time so I am at a bit of a loss where there seems to be no correlation.  

Sean.

-----Original Message-----
From: Peter Manev <petermanev at gmail.com> 
Sent: Thursday, February 21, 2019 8:20 AM
To: Cloherty, Sean E <scloherty at mitre.org>
Cc: Eric Urban <eurban at umn.edu>; Open Information Security Foundation <oisf-users at lists.openinfosecfoundation.org>
Subject: Re: [EXT] Re: [Oisf-users] Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

On Wed, Feb 20, 2019 at 7:27 PM Cloherty, Sean E <scloherty at mitre.org> wrote:
>
> Done.
>
> Upon restart of Suricata I Immediately observed the behavior I had mentioned before where the " SURICATA TLS invalid heartbeat encountered, possible exploit attempt (heartbleed)" starts firing very frequently for what I think are false positives.
>
> The 4.1.2 server will detect the dozen or so daily hits which are recorded on the production host with the same traffic being analyzed.  But the bulk of the alerts, the ones which I believe are FPs, are all from traffic between users connected to our network via VPN and our proxy servers.
>

Ok - your drop rate is not hugely different between 4.1.2 and 4.0.6 though right or?

> -----Original Message-----
> From: Peter Manev <petermanev at gmail.com>
> Sent: Wednesday, February 20, 2019 12:26 PM
> To: Cloherty, Sean E <scloherty at mitre.org>
> Cc: Eric Urban <eurban at umn.edu>; Open Information Security Foundation 
> <oisf-users at lists.openinfosecfoundation.org>
> Subject: Re: [EXT] Re: [Oisf-users] Packet loss and increased resource 
> consumption after upgrade to 4.1.2 with Rust support
>
> On Wed, Feb 20, 2019 at 3:31 PM Cloherty, Sean E <scloherty at mitre.org> wrote:
> >
> > So do this with 4.1.2, right?
>
> Yes please!
>
> >
> > -----Original Message-----
> > From: Peter Manev <petermanev at gmail.com>
> > Sent: Wednesday, February 20, 2019 9:27 AM
> > To: Cloherty, Sean E <scloherty at mitre.org>
> > Cc: Eric Urban <eurban at umn.edu>; Open Information Security 
> > Foundation <oisf-users at lists.openinfosecfoundation.org>
> > Subject: Re: [EXT] Re: [Oisf-users] Packet loss and increased 
> > resource consumption after upgrade to 4.1.2 with Rust support
> >
> > On Wed, Feb 20, 2019 at 3:01 PM Cloherty, Sean E <scloherty at mitre.org> wrote:
> > >
> > > Peter -
> > >
> > > * CentOS Linux release 7.6.1810 / 3.10.0-957.1.3.el7.x86_64 #1 SMP 
> > > Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
> > > * Rust is rust-1.30.0-x86_64
> > > * Hyperscan 4.7.0
> > > *Here is the full line I use to build Suricata -
> > >
> > > ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var 
> > > --with-libhs-libraries=/usr/local/lib/
> > > --with-libhs-includes=/usr/local/include/hs/
> > > --with-liblzma-includes=/usr/include/lzma/
> > > --with-liblzma-libraries=/usr/lib64/ --enable-gccprotect 
> > > --enable-geoip --enable-lua --enable-profiling --enable-pie 
> > > --enable-rust --enable-unix-socket
> > >
> >
> > Can you remove "--enable-profiling" form the command line compile, reinstall  and rerun the test? (--enable-rust --enable-unix-socket should be done/checked  by default during build time) Curious if you could please  do that and feedback the results  with all Rust/protos enabled?
> >
> > Thank you
> >
> > > Thanks,
> > >
> > > Sean
> > >
> > > -----Original Message-----
> > > From: Peter Manev <petermanev at gmail.com>
> > > Sent: Wednesday, February 20, 2019 5:59 AM
> > > To: Cloherty, Sean E <scloherty at mitre.org>
> > > Cc: Eric Urban <eurban at umn.edu>; Open Information Security 
> > > Foundation <oisf-users at lists.openinfosecfoundation.org>
> > > Subject: Re: [EXT] Re: [Oisf-users] Packet loss and increased 
> > > resource consumption after upgrade to 4.1.2 with Rust support
> > >
> > > On Tue, Feb 19, 2019 at 1:12 AM Cloherty, Sean E <scloherty at mitre.org> wrote:
> > > >
> > > > When I compiled 4.0.6 on the previously 4.1.2 host, I used the same arguments including Rust.  I think all of the Rust parsers are disabled, but SMB is enabled.  In my case I’ve seen no packet loss in three days despite compiling with Rust.
> > > >
> > > >
> > >
> > > Interesting observation and feedback - thank you Sean Sould you please confirm what OS / rust version do you use?
> > >
> > > Thank you
> > >
> > >
> > > --
> > > Regards,
> > > Peter Manev
> >
> >
> >
> > --
> > Regards,
> > Peter Manev
>
>
>
> --
> Regards,
> Peter Manev

--
Regards,
Peter Manev
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 412stats.txt
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190221/9276a302/attachment-0001.txt>