[Oisf-users] Packet loss and increased resource consumption after upgrade to 4.1.2 with Rust support

[Peter Manev]

On Tue, Feb 5, 2019 at 11:14 PM Eric Urban <eurban at umn.edu> wrote:
>
> I have seen a few emails on this list about users either having packet loss or increased resource consumption after upgrading to 4.1.2.  We are seeing much higher rates of packet loss after upgrading to 4.1.2 (with Rust) from 4.0.6 (no Rust) so would appreciate any input on how to best move forward with troubleshooting.  Please let me know if it would it be better to open a ticket in Redmine.
>
> Here are some details:
> - We have two sets of Suricata sensors that are each getting the same set of traffic, so one acts as a redundant set.  These have the same hardware.
> - Once we upgraded to 4.1.2, cpu and memory usage went up and we have had regular bursts of heavy packet loss.  I sampled traffic from yesterday early morning through today and a few sensors have had 2.49, 9.36, and 11.130% packet loss over that time frame.  For our 4.0.6 sensor set over the same time for the same traffic the sensor with the highest loss has 0.011%.  We have also had one occasion where a sensor had possible memory exhaustion as the stats.tcp.ssn_memcap_drop_delta counter hit 199.
> - We rolled back our primary sensor set to 4.0.6 and immediately stopped having drops.
> - We did not explicitly enable or disable any of the Rust parsers in our config (krb5, nfs, tftp, ntp, dhcp, ikev2) but do have SMB enabled so I believe will have the SMB2/3 parser.  I was not sure the default behavior in this case (as --dump-config had no values for the new Rust based parsers), so I did test disabling krb5, nfs, tftp, ntp, dhcp, and ikev2.  We still had high percentages of drops in this case.  I plan to look into whether or not there is a way to disable just SMB2/3 with Rust enabled to see if that makes a difference.
> - We use pcap capture mode with Myricom cards.  The driver version if not at the latest, though is only one patch version away from the latest.  We tested updating to the latest version on one of our sensors and it had no effect.
> - Suricata was compiled with rustc 1.30.1.  I did try upgrading to use Rust 1.31 but did not seem to have any effect.
> - I compiled Suricata 4.1.2 without Rust and that looks to have positively affected this.  We had very little packet loss in this case.
>

Is it possible to share full stats.log from the two different runs
(4.1.2 and 4.0.6) and any changes made to suricata.yaml ?


-- 
Regards,
Peter Manev