[Oisf-users] Reduce memory consumption - low hanging fruits?

[Timo Sigurdsson]

Hi Andreas,

Andreas Herz schrieb am 11.04.2020 21:13 (GMT +02:00):
> Hi Timo
> 
> On 11/04/20 at 02:37, Timo Sigurdsson wrote:
>> I use Suricata 4.1.2 on Debian 10 in IDS mode. It runs pretty well,
>> but occasionally (like every 4-5 weeks), the memory consumption grows
>> out of hand. My firewall system only has 4GB of RAM and cannot be
>> extended (PC Engines APU2). Therefore, I would like to tweak the
>> configuration to reduce the memory consumption a bit. I already
>> started by shrinking my ruleset down from ~23000 rules to ~15500. That
>> didn't seem to do much. After starting Suricata, it uses a few hundred
>> megabytes of RAM (200-300MB). This slowly increases to about
>> 800-900MB, but then once in a while there are spikes hitting +3GB. I
>> have a memory limit defined in the systemd unit for suricata (3.2GB),
>> so once it exceeds that, Suricata gets killed and restarted. I can't
>> really make out when exactly this happens, but I assume it's when I
>> have multiple users maxing out the available bandwidth with streaming,
>> downloads and so on. The connection is a PPPoE connection with
>> 85MBit/s down and 35Mbit/s up.
> 
> I would update to the latest version, ideally switch to v5. I don't see
> any huge potential in your configuration as well. Although I don't see
> NFLOG used much, so could be a bug as well. When you have a suspicion of
> the problematic traffic, try to reproduce it and take a look into the
> stats.log especially the memcaps.
> 
>> Does memory usage differ among different runmodes? I currently use
>> nflog but would it be beneficial to use afpacket instead? One of the
>> reasons I like nflog is that I don't need to pass the packets that are
>> dropped anyway to Suricata.
> 
> It could, it might be worth a try to just use af-packet mode to
> double-check if it's related to the method.
> 
> -- 
> Andreas Herz

thanks. I built a backport of 5.0.2 on Debian stable and am running it now. So far, it seems to behave better in terms of memory use, but I'll have to see how it turns out in the long run. (After 2 days it uses only about ~350MB. With 4.1.2 I just had to wait for an hour or two until the memory use was up at 800-900MB.) I did see one warning once that I haven't had before, but it only happened once after a reboot and I couldn't reproduce it (tried restarting suricata and rebooting the system several times). It was this message and was logged immediately after the engine and packet processing threads were initialized:
<Warning> -- [ERRCODE: SC_ERR_NFLOG_HANDLE_PKT(252)] - nflog_handle_packet error -1

I'll have to see whether that shows up again anytime soon. Maybe it was just an odd race condition, who knows. In any case, I will keep an eye on the memory consumption (and any other warnings/errors that may come up), but it will likely take a while to draw final conclusions given that it usually took weeks with the older version to see that the memory limit was exceeded. I did, however, enable the stats log which I had previously disabled, so I can have a look at if the problem occurs again, as suggested.

As for testing the af_packet runmode, I will take a look at it if the problem arises again, but if not, I prefer to stick with my current setup.

Btw., while we are at it, I just commented on an old feature request for a Suricata Debian repository since the "workaround" mentioned in this ticket (the backports repository) doesn't seem to be maintained/updated anymore: https://redmine.openinfosecfoundation.org/issues/1216

Regards,

Timo