[Oisf-devel] Live rule swap testing

[Victor Julien]

On 07/03/2012 04:33 PM, Martin Holste wrote:
> Now the third time I do the rule reload, nothing gets stuck, but the
> RAM is never given back:
> 
> 10963 root      20   0 36.3g  35g  67m S  464 25.3  14:42.75 suricata
> 10963 root      20   0 73.4g  70g  67m S  616 50.0  79:26.96 suricata
> 10963 root      20   0  110g 105g  67m S  481 74.4 114:36.57 suricata

Did you try with tcmalloc as well?

Cheers,
Victor

> On Tue, Jul 3, 2012 at 9:00 AM, Martin Holste <mcholste at gmail.com> wrote:
>> Ok, giving this a shot.  Also, I should note that since build
>> e3764b90c3bdfffaf8d98ec5bd71543a37b3f407 (or around there in early
>> June) Suricata crashes regularly.  I'll start running under gdb to
>> hopefully get you the backtrace (can't do core dumps, they're too
>> large).
>>
>> On Tue, Jul 3, 2012 at 6:33 AM, Victor Julien <victor at inliniac.net> wrote:
>>> We've fixed the biggest leaks in the current git master. Some memory
>>> still gets lost it seems, probably due to fragmentation. Dealing with
>>> this will be a 1.4 effort.
>>>
>>> With 15k rules I now get:
>>>
>>> 17691 root      20   0 1973m 1.3g 3532 S 35.9 17.0   0:44.32 lt-suricata
>>> 17691 root      20   0 2945m 2.2g 3756 S 41.9 28.6   1:51.01 lt-suricata
>>> 17691 root      20   0 3335m 2.6g 3756 S 42.5 33.8   2:55.43 lt-suricata
>>> 17691 root      20   0 3343m 2.6g 3756 S 41.5 33.8   4:08.54 lt-suricata
>>> 17691 root      20   0 3351m 2.6g 3756 S 38.2 33.9   5:12.40 lt-suricata
>>>
>>> First entry is cold start, every one after is another reload. It's clear
>>> that mem usage stabilizes at some point.
>>>
>>> Interestingly, using tcmalloc gives a lot better results for the exact
>>> same ruleset:
>>>
>>> 17865 root      20   0 1418m 1.2g 3964 S 37.2 16.0   0:36.16 lt-suricata
>>> 17865 root      20   0 2194m 2.0g 4016 S 36.9 25.6   1:30.87 lt-suricata
>>> 17865 root      20   0 2509m 2.0g 4088 S 37.5 26.0   2:28.65 lt-suricata
>>> 17865 root      20   0 2517m 2.0g 4088 S 44.2 26.1   3:24.68 lt-suricata
>>>
>>> I've added a tcmalloc page to the performance section of the wiki:
>>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Tcmalloc
>>>
>>> On 06/30/2012 04:15 PM, Martin Holste wrote:
>>>> Great, thanks.
>>>>
>>>> On Sat, Jun 30, 2012 at 4:16 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>> On 06/29/2012 09:44 PM, Martin Holste wrote:
>>>>>> Suricata's not giving back the RAM after doing the live swap, which is
>>>>>> a big (99G in my case) problem:
>>>>>>
>>>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>  8513 root      20   0 99.7g  96g  67m S  445 68.3  36:01.06 suricata
>>>>>>
>>>>>> It also appears to do this for each live swap, so by the third swap, I'm OOM.
>>>>>>
>>>>>> This is on the latest from this morning, commit:
>>>>>> 4cf6bb3f4cbdab8a0cd57964be801cf676d2ec26.
>>>>>>
>>>>>> Suggestions?
>>>>>
>>>>> Seeing the same. We're tracking the issue here:
>>>>> https://redmine.openinfosecfoundation.org/issues/492
>>>>>
>>>>> --
>>>>> ---------------------------------------------
>>>>> Victor Julien
>>>>> http://www.inliniac.net/
>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>> ---------------------------------------------
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> ---------------------------------------------
>>> Victor Julien
>>> http://www.inliniac.net/
>>> PGP: http://www.inliniac.net/victorjulien.asc
>>> ---------------------------------------------
>>>
>>>
>>>
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------