[Oisf-devel] Live rule swap testing

Tue Jul 3 14:33:09 UTC 2012

Now the third time I do the rule reload, nothing gets stuck, but the
RAM is never given back:

10963 root      20   0 36.3g  35g  67m S  464 25.3  14:42.75 suricata
10963 root      20   0 73.4g  70g  67m S  616 50.0  79:26.96 suricata
10963 root      20   0  110g 105g  67m S  481 74.4 114:36.57 suricata

On Tue, Jul 3, 2012 at 9:00 AM, Martin Holste <mcholste at gmail.com> wrote:
> Ok, giving this a shot.  Also, I should note that since build
> e3764b90c3bdfffaf8d98ec5bd71543a37b3f407 (or around there in early
> June) Suricata crashes regularly.  I'll start running under gdb to
> hopefully get you the backtrace (can't do core dumps, they're too
> large).
>
> On Tue, Jul 3, 2012 at 6:33 AM, Victor Julien <victor at inliniac.net> wrote:
>> We've fixed the biggest leaks in the current git master. Some memory
>> still gets lost it seems, probably due to fragmentation. Dealing with
>> this will be a 1.4 effort.
>>
>> With 15k rules I now get:
>>
>> 17691 root      20   0 1973m 1.3g 3532 S 35.9 17.0   0:44.32 lt-suricata
>> 17691 root      20   0 2945m 2.2g 3756 S 41.9 28.6   1:51.01 lt-suricata
>> 17691 root      20   0 3335m 2.6g 3756 S 42.5 33.8   2:55.43 lt-suricata
>> 17691 root      20   0 3343m 2.6g 3756 S 41.5 33.8   4:08.54 lt-suricata
>> 17691 root      20   0 3351m 2.6g 3756 S 38.2 33.9   5:12.40 lt-suricata
>>
>> First entry is cold start, every one after is another reload. It's clear
>> that mem usage stabilizes at some point.
>>
>> Interestingly, using tcmalloc gives a lot better results for the exact
>> same ruleset:
>>
>> 17865 root      20   0 1418m 1.2g 3964 S 37.2 16.0   0:36.16 lt-suricata
>> 17865 root      20   0 2194m 2.0g 4016 S 36.9 25.6   1:30.87 lt-suricata
>> 17865 root      20   0 2509m 2.0g 4088 S 37.5 26.0   2:28.65 lt-suricata
>> 17865 root      20   0 2517m 2.0g 4088 S 44.2 26.1   3:24.68 lt-suricata
>>
>> I've added a tcmalloc page to the performance section of the wiki:
>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Tcmalloc
>>
>> On 06/30/2012 04:15 PM, Martin Holste wrote:
>>> Great, thanks.
>>>
>>> On Sat, Jun 30, 2012 at 4:16 AM, Victor Julien <victor at inliniac.net> wrote:
>>>> On 06/29/2012 09:44 PM, Martin Holste wrote:
>>>>> Suricata's not giving back the RAM after doing the live swap, which is
>>>>> a big (99G in my case) problem:
>>>>>
>>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>  8513 root      20   0 99.7g  96g  67m S  445 68.3  36:01.06 suricata
>>>>>
>>>>> It also appears to do this for each live swap, so by the third swap, I'm OOM.
>>>>>
>>>>> This is on the latest from this morning, commit:
>>>>> 4cf6bb3f4cbdab8a0cd57964be801cf676d2ec26.
>>>>>
>>>>> Suggestions?
>>>>
>>>> Seeing the same. We're tracking the issue here:
>>>> https://redmine.openinfosecfoundation.org/issues/492
>>>>
>>>> --
>>>> ---------------------------------------------
>>>> Victor Julien
>>>> http://www.inliniac.net/
>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>> ---------------------------------------------
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Oisf-devel mailing list
>>>> Oisf-devel at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>
>>>
>>>
>>
>>
>> --
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>>
>>