[Oisf-users] Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Eric Urban]

Hello Peter,

I sent over the stats logs to you directly.  For your question about our
config, there are no differences between 4.0.6 and 4.1.2.  Are you looking
for me to send over our config file or is my answer of them not having
differences sufficient?

Thank you,
Eric


On Wed, Feb 6, 2019 at 10:54 AM Peter Manev <petermanev at gmail.com> wrote:

> On Tue, Feb 5, 2019 at 11:14 PM Eric Urban <eurban at umn.edu> wrote:
> >
> > I have seen a few emails on this list about users either having packet
> loss or increased resource consumption after upgrading to 4.1.2.  We are
> seeing much higher rates of packet loss after upgrading to 4.1.2 (with
> Rust) from 4.0.6 (no Rust) so would appreciate any input on how to best
> move forward with troubleshooting.  Please let me know if it would it be
> better to open a ticket in Redmine.
> >
> > Here are some details:
> > - We have two sets of Suricata sensors that are each getting the same
> set of traffic, so one acts as a redundant set.  These have the same
> hardware.
> > - Once we upgraded to 4.1.2, cpu and memory usage went up and we have
> had regular bursts of heavy packet loss.  I sampled traffic from yesterday
> early morning through today and a few sensors have had 2.49, 9.36, and
> 11.130% packet loss over that time frame.  For our 4.0.6 sensor set over
> the same time for the same traffic the sensor with the highest loss has
> 0.011%.  We have also had one occasion where a sensor had possible memory
> exhaustion as the stats.tcp.ssn_memcap_drop_delta counter hit 199.
> > - We rolled back our primary sensor set to 4.0.6 and immediately stopped
> having drops.
> > - We did not explicitly enable or disable any of the Rust parsers in our
> config (krb5, nfs, tftp, ntp, dhcp, ikev2) but do have SMB enabled so I
> believe will have the SMB2/3 parser.  I was not sure the default behavior
> in this case (as --dump-config had no values for the new Rust based
> parsers), so I did test disabling krb5, nfs, tftp, ntp, dhcp, and ikev2.
> We still had high percentages of drops in this case.  I plan to look into
> whether or not there is a way to disable just SMB2/3 with Rust enabled to
> see if that makes a difference.
> > - We use pcap capture mode with Myricom cards.  The driver version if
> not at the latest, though is only one patch version away from the latest.
> We tested updating to the latest version on one of our sensors and it had
> no effect.
> > - Suricata was compiled with rustc 1.30.1.  I did try upgrading to use
> Rust 1.31 but did not seem to have any effect.
> > - I compiled Suricata 4.1.2 without Rust and that looks to have
> positively affected this.  We had very little packet loss in this case.
> >
>
> Is it possible to share full stats.log from the two different runs
> (4.1.2 and 4.0.6) and any changes made to suricata.yaml ?
>
>
> --
> Regards,
> Peter Manev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190206/373c75c6/attachment-0001.html>