[Oisf-users] Suricata causes massive packet loss

[Peter Manev]

On Tue, Sep 3, 2019 at 7:13 PM Nelson, Cooper <cnelson at ucsd.edu> wrote:
>
> Hi Peter,
>
> It's been a few years since I've done an inline deployment, so I'll see what I can do.  I can say I did get it working 100% in a proxy configuration (listening on the inside interface of a squid proxy).
>
> First of all, as a sanity check I'll suggest setting up an "R&D" box running the latest build from git and latest stable linux kernel.  There may be a kernel/driver/library/suricata bug that has since been addressed.  If that turns out to be the case, suricata 5.0 is in beta so the stable release should not be that far away.
>
> Assuming you started here ...
>
> https://suricata.readthedocs.io/en/latest/setting-up-ipsinline-for-linux.html?highlight=inline
>
> ... make sure you are using the 'autofp' runmode.   If you are using AF_PACKET with tpacket-v3 set the block size/timeout to a low value and start with a single worker thread for testing.  Also, try using libpcap instead of AF_PACKET, I'm pretty sure that's what I ended up using and the performance was fine for a 1gbit deployment.
>
> -Coop
>

Hi Peter,

I also noticed  a couple of things that i think could be tried out to
see if they improve the situation:

- use rust, some of the major protos (like dns) have rust implementation
- use libhtp v0.5.30 , noticed your current one is v0.5.28

What exactly is your setup/start commands etc?
I noticed in your log -
tcp.pkt_on_wrong_thread                       | Total
   | 30622

Thank you

> -----Original Message-----
> From: Oisf-users <oisf-users-bounces at lists.openinfosecfoundation.org> On Behalf Of peter.mueller at ipfire.org
> Sent: Tuesday, September 3, 2019 6:41 AM
> To: oisf-users at lists.openinfosecfoundation.org
> Cc: IPFire: Development-List <development at lists.ipfire.org>
> Subject: [Oisf-users] Suricata causes massive packet loss
>
> Dear OSIF/Suricata users,
>
> earlier this year, the linux-based open source firewall distribution IPFire has migrated from Snort to Suricata for a number of reasons (further information is available at https://blog.ipfire.org/post/introducing-ipfire-s-new-intrusion-prevention-system).
>
> While we are quite pleased with some of its features (multi-threading, ability to monitor several interfaces per process, etc.), we experienced some problems ever since we are running it. Not being reproducible everywhere, we initially thought they were corner cases in obscure network scenarios.
>
> Ultimately, they were not. Even worse, no dropped packets were logged although we can tell for sure there were some.
>
> For example, several IPFire users - including myself - report very slow DNS resolution when trying to access a website, while "normal" lookups using dig or host commands perform fine.
>
> Another issue is reduced OpenVPN tunnel throughput, which seems to be caused by massive packet loss when Suricata is enabled (~ 800 kB/s, ~ 2 MB/s if Suricata is turned off). In order to get closer to its origin, we spend a lot of time on testing and debugging, eventually left without any idea what the solution might be.
>
> Both issues - possibly being related to each other - can be reproduced using Suricata 4.1.4 without any rules or packet decoders enabled. Unfortunately, our setup, where Suricata runs inline via Netfilter queue, does not seem to be documented very well.
>
> That's why I am asking here if anybody is able to tell us what we are doing wrong. Perhaps this just might be a configuration problem, but we are out of ideas where to look for it.
>
> Please find our suricata.yaml (decoders enabled, but disabling it does not matter) and the stats.log file enclosed.
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/



-- 
Regards,
Peter Manev