[Oisf-users] USR2 causes increased memory usage causing fast.log to stop appending

[Victor Julien]

On 13-04-16 02:36, Zeolla at GMail.com wrote:
> I recently upgraded my boxes to 3.0.1, and I seem to be having stability
> issues when I send a USR2 after each day's signature update.  Memory
> consumption seems to skyrocket after the USR2 (similar to when first
> starting the server), however my boxes are currently running with very
> little room to spare.  Due to the comments in this issue
> <https://redmine.openinfosecfoundation.org/issues/1659>, I'm not clear
> if it is currently required to duplicate in-memory rules after receiving
> a USR2 to do the update (and this would be a feature request), or if
> this is a true issue.  

When a reload is performed a new detection engine is allocated while the
old one stays active. When the loading of the new detect engine is
complete, the engines are hotswapped. After this the old engine is freed.

The memory required for the detection engines will be twice the size of
a single engine during the reload.


> My fast.log stops getting written to, but the service and PID stay
> alive, making it somewhat difficult to monitor for and automatically fix
> with something like a puppet service ensure => running.  In addition, if
> I don't do a service restart or USR2 for more than a day, I see the same
> issue where fast.log stops appending but the service/pid stays up.  
> 
> I have been able to verify that if I send a USR2 not long after a
> service restart (when there is plenty of RAM to spare), it will
> successfully complete the rule load and fast.log will continue to work. 
> If I then send a second or third USR2, memory usage does not change
> significantly.  
> 
> In addition, the issue I mentioned earlier was resolved with comments
> that I don't fully understand.  Was the core issue supposed to be fixed,
> or just the memory leaks?  

We fixed a lot of leaks that would lead to increased memory use with
every reload. We do still see increased memory use after the first
reload, but after that it should be stable.


> Any other suggestions?  I will be looking into compiling a debugging
> version of my package with LeakSanitizer in the near future (probably
> tomorrow).  

LeakSanitizer output would be very welcome. We've fixed many leaks in
3.0.1, some small, some fairly big, but it's possible there are more.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------