[Oisf-users] suricata memory leak?

[Peter Manev]

On Fri, Jan 18, 2019 at 2:04 PM Michael Stone <mstone at mathom.us> wrote:
>
> On Fri, Jan 18, 2019 at 09:13:35AM +0100, you wrote:
> >On Thu, Jan 17, 2019 at 3:36 PM Michael Stone <mstone at mathom.us> wrote:
> >>
> >> On Thu, Jan 17, 2019 at 02:15:38PM +0100, you wrote:
> >> >On Thu, Jan 17, 2019 at 2:04 PM Michael Stone <mstone at mathom.us> wrote:
> >> >>
> >> >> Are other people noticing much higher memory consumption in suricata
> >> >> 4.1? As an example, one instance of 4.1.2 I started yesterday grew from
> >> >> 7G memory used to 21G overnight. There's nothing in the memory stats
> >> >> output to account for the change, so I'm not sure where the memory is
> >> >> going.
> >> >
> >
> >Is it a case where it is still at 21G all the time or it peaked to 21G
> >and then returned to lower value?
>
> It typically grows unbounded unless linux OOM starts killing pieces.
> That particular system has 32G RAM so it hits the wall a lot sooner, but
> the ones with 100+G RAM just have a longer runway to grow. I've got
> dozens of sensors deployed, and they grow at different rates. Some are
> much, much smaller and aren't growing much at all (but they see almost
> no traffic). OTOH, I've got two 10gbps sensors that I restarted suricata
> on yesterday, and one is at 15GB RAM and the other is at 27GB RAM. Same
> initial config, so I guess it's somehow related to traffic, but I don't
> know what. The sensor I talked about initially, that went from 7 to 21G,
> was restarted yesterday and today is still at 7 after the same amount of
> runtime on the same network. The inconsistency is part of why it's taken

So it appears that some sensors have the issue some dont but the ones
that have it are displaying the behavior inconsistently, correct ?

> me so long to really start pursuing it, but I've become pretty well
> convinced there's been a fundamental change in behavior with 4.1.
>

If you compare to 4.0 - yes there have been quite a few new
features/additions/changes .


> At any rate, I'm guessing from your replies that this isn't
> widespread/known? I'm currently working on building with a newer version

I haven noticed it and I just realized that i had cpl of big sensors
that are basically runing 4.1.2 for a month now without cold restarts
with no issue so far (that i have noticed anyway, doesn't meant there
could be no issue though :) )

> of rust, as it's the biggest change I can think of between 4.0 and 4.1,
> and I've got some sensors that were running for years before showing
> these memory issues with 4.1.
>
> >> >How much traffic are you inspecting and what is the output (to begin with) of
> >> >suricata --dump-config |grep mem
> >> >?
> >>
> >> Not particularly high traffic at this sensor at a branch office, about
> >> 25Mbps daytime average and 200Mbps during overnight backups. Seeing the
> >> same problem on high and low volume links.
> >>
> >> defrag.memcap = 256mb
> >> flow.memcap = 512mb
> >> stream.memcap = 128mb
> >> stream.reassembly.memcap = 128mb
> >> host.memcap = 16777216
> >>
> >

That sensor(config) is the one that goes up to 21G consumption sometimes ?

> >How do you  start Suricata  and how do you reload rules ?
>
> /usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /var/run/suricata.pid
>
> suricatasc -c reload-rules
>
> It's not the case that the memory consumption jumps immediately after a
> reload.


So the sensors that exhibit this behavior  - have the ruleset reloaded
, not Suricata restarted - right ?

Thank you


-- 
Regards,
Peter Manev