[Oisf-devel] Live rule swap testing

Tue Jul 3 15:04:02 UTC 2012

On 07/03/2012 04:56 PM, Martin Holste wrote:
> Nope, no tcmalloc yet (have to compile from source on my platform) but
> should that really be what's needed to eliminate the full leak?

Almost all leaks have been fixed, especially all larger ones. The issue
here is very likely memory fragmentation. We've been planning to address
that for some time now, and will in the next dev cycle. Until then the
reload feature will be labeled "experimental", or in other words YMMV.

tcmalloc has been shown to prevent some of this fragmentation, so it may
help.

Cheers,
Victor

> I'll try to run under gdb tonight.
> 
> On Tue, Jul 3, 2012 at 9:55 AM, Victor Julien <victor at inliniac.net> wrote:
>> On 07/03/2012 04:00 PM, Martin Holste wrote:
>>> Ok, giving this a shot.  Also, I should note that since build
>>> e3764b90c3bdfffaf8d98ec5bd71543a37b3f407 (or around there in early
>>> June) Suricata crashes regularly.  I'll start running under gdb to
>>> hopefully get you the backtrace (can't do core dumps, they're too
>>> large).
>>
>> bt's highly appreciated. Can't fix issues we don't know about :)
>>
>> Cheers,
>> Victor
>>
>>> On Tue, Jul 3, 2012 at 6:33 AM, Victor Julien <victor at inliniac.net> wrote:
>>>> We've fixed the biggest leaks in the current git master. Some memory
>>>> still gets lost it seems, probably due to fragmentation. Dealing with
>>>> this will be a 1.4 effort.
>>>>
>>>> With 15k rules I now get:
>>>>
>>>> 17691 root      20   0 1973m 1.3g 3532 S 35.9 17.0   0:44.32 lt-suricata
>>>> 17691 root      20   0 2945m 2.2g 3756 S 41.9 28.6   1:51.01 lt-suricata
>>>> 17691 root      20   0 3335m 2.6g 3756 S 42.5 33.8   2:55.43 lt-suricata
>>>> 17691 root      20   0 3343m 2.6g 3756 S 41.5 33.8   4:08.54 lt-suricata
>>>> 17691 root      20   0 3351m 2.6g 3756 S 38.2 33.9   5:12.40 lt-suricata
>>>>
>>>> First entry is cold start, every one after is another reload. It's clear
>>>> that mem usage stabilizes at some point.
>>>>
>>>> Interestingly, using tcmalloc gives a lot better results for the exact
>>>> same ruleset:
>>>>
>>>> 17865 root      20   0 1418m 1.2g 3964 S 37.2 16.0   0:36.16 lt-suricata
>>>> 17865 root      20   0 2194m 2.0g 4016 S 36.9 25.6   1:30.87 lt-suricata
>>>> 17865 root      20   0 2509m 2.0g 4088 S 37.5 26.0   2:28.65 lt-suricata
>>>> 17865 root      20   0 2517m 2.0g 4088 S 44.2 26.1   3:24.68 lt-suricata
>>>>
>>>> I've added a tcmalloc page to the performance section of the wiki:
>>>> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Tcmalloc
>>>>
>>>> On 06/30/2012 04:15 PM, Martin Holste wrote:
>>>>> Great, thanks.
>>>>>
>>>>> On Sat, Jun 30, 2012 at 4:16 AM, Victor Julien <victor at inliniac.net> wrote:
>>>>>> On 06/29/2012 09:44 PM, Martin Holste wrote:
>>>>>>> Suricata's not giving back the RAM after doing the live swap, which is
>>>>>>> a big (99G in my case) problem:
>>>>>>>
>>>>>>>  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
>>>>>>>  8513 root      20   0 99.7g  96g  67m S  445 68.3  36:01.06 suricata
>>>>>>>
>>>>>>> It also appears to do this for each live swap, so by the third swap, I'm OOM.
>>>>>>>
>>>>>>> This is on the latest from this morning, commit:
>>>>>>> 4cf6bb3f4cbdab8a0cd57964be801cf676d2ec26.
>>>>>>>
>>>>>>> Suggestions?
>>>>>>
>>>>>> Seeing the same. We're tracking the issue here:
>>>>>> https://redmine.openinfosecfoundation.org/issues/492
>>>>>>
>>>>>> --
>>>>>> ---------------------------------------------
>>>>>> Victor Julien
>>>>>> http://www.inliniac.net/
>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>> ---------------------------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Oisf-devel mailing list
>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> ---------------------------------------------
>>>> Victor Julien
>>>> http://www.inliniac.net/
>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>> ---------------------------------------------
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>>
>>
>> _______________________________________________
>> Oisf-devel mailing list
>> Oisf-devel at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> 

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------