[Oisf-users] Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Eric Urban]

I have seen a few emails on this list about users either having packet loss
or increased resource consumption after upgrading to 4.1.2.  We are seeing
much higher rates of packet loss after upgrading to 4.1.2 (with Rust) from
4.0.6 (no Rust) so would appreciate any input on how to best move forward
with troubleshooting.  Please let me know if it would it be better to open
a ticket in Redmine.

Here are some details:
- We have two sets of Suricata sensors that are each getting the same set
of traffic, so one acts as a redundant set.  These have the same hardware.
- Once we upgraded to 4.1.2, cpu and memory usage went up and we have had
regular bursts of heavy packet loss.  I sampled traffic from yesterday
early morning through today and a few sensors have had 2.49, 9.36, and
11.130% packet loss over that time frame.  For our 4.0.6 sensor set over
the same time for the same traffic the sensor with the highest loss has
0.011%.  We have also had one occasion where a sensor had possible memory
exhaustion as the stats.tcp.ssn_memcap_drop_delta counter hit 199.
- We rolled back our primary sensor set to 4.0.6 and immediately stopped
having drops.
- We did not explicitly enable or disable any of the Rust parsers in our
config (krb5, nfs, tftp, ntp, dhcp, ikev2) but do have SMB enabled so I
believe will have the SMB2/3 parser.  I was not sure the default behavior
in this case (as --dump-config had no values for the new Rust based
parsers), so I did test disabling krb5, nfs, tftp, ntp, dhcp, and ikev2.
We still had high percentages of drops in this case.  I plan to look into
whether or not there is a way to disable just SMB2/3 with Rust enabled to
see if that makes a difference.
- We use pcap capture mode with Myricom cards.  The driver version if not
at the latest, though is only one patch version away from the latest.  We
tested updating to the latest version on one of our sensors and it had no
effect.
- Suricata was compiled with rustc 1.30.1.  I did try upgrading to use Rust
1.31 but did not seem to have any effect.
- I compiled Suricata 4.1.2 without Rust and that looks to have positively
affected this.  We had very little packet loss in this case.


-- 
Eric Urban
University Information Security | Office of Information Technology |
it.umn.edu
University of Minnesota | umn.edu
eurban at umn.edu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20190205/88f136c9/attachment.html>