[Oisf-users] Suricata v2.1beta2 with geoip and high ram consumption

[Jay M.]

On Thu, Jan 1, 2015 at 10:15 AM, Peter Manev <petermanev at gmail.com> wrote:
> On Wed, Dec 31, 2014 at 4:13 PM, Jay M. <jskier at gmail.com> wrote:
>> I've been playing around a little with a geoip rule and noticed only
>> when the sole one is enabled, ram is gobbled up quickly (about an
>> hour) and eats into the swap with 16 gigs of ram.
>>
>
> What is the sum total of all your mem settings in suricata.yaml?

About 16.3 GB if the host memcap is kilobytes. Everything else is
commented out / default. I am hashing all and do store some files,
usually a handful a day.

degrag memcap: 32mb
flow memcap: 64mb
stream memcap: 64mb
stream reassembly: 128 mb
host memcap: 16777216 (16 GB?)

I have mitigated the eating in to swap problem for now by changing my
rule update script to run every 6 hours and restart the daemon as
opposed to reloading it (see the other caveat below). I read in the
wiki that rule reloading is still in a delicate state, so this makes
sense.

>
>> So, I've added more RAM to the VM, from 16 to 24 gigs, I'll see what
>> that does (up to 15 gigs allocated after starting 40 minutes ago).
>>
>> It does not appear to be dropping packets and the rule is working, as
>> well as the ETPRO set. I'm wondering if others using geo rules are
>> also seeing this behavior? I'm not ready to call it a memory leak just
>> yet...
>
> What amount of traffic are you inspecting?
> Is this reproducible only (and every time) when you enable geoip?

I am inspecting a 100 meg pipe using rspan, and am monitoring only. On
my virtual host box in VMware 11, I passthru a poor man receiver so to
speak, which is a 1 gig USB3 dongle. Not the most ideal setup I know,
but it actually works fairly well and should hold me off until erspan
span gets implemented in suricata.

RAM consumption is quickly reproducible with the one geoip rule
(basically if not US, alert) although there is another gothca I'm
looking into. I noticed my script to reload the rules every four hours
by invoking the kill command (as noted in the wiki) via a systemd unit
also will eat up a lot of RAM (usually 3~4 gig chunks per reload),
albeit noticeably fewer volume gobbled in time than the geoip rule. I
noticed after a weekend before the geoip rule was deployed this
basically killed suricata because it it ate up all the ram and swap
when I was at 16/8 ram/swap respectively.

>>
>> Additionally, running 64-bit, ArchLinux 3.17.6 kernel.
>>
>> --
>> Jay
>> jskier at gmail.com
>> _______________________________________________
>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
>> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
>> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> Training now available: http://suricata-ids.org/training/
>
>
>
> --
> Regards,
> Peter Manev

--
Jay
jskier at gmail.com