[Oisf-devel] Memory pb on Suricata git today
rmkml
rmkml at free.fr
Sat Jul 31 17:32:34 UTC 2010
It's work with low profile on detect-engine (for information, with this low profile, suricata use 940M of memory with my personnal ruleset).
Big thx you Victor!
Im back to Suricata testing...
Best Regards
Rmkml
On Sat, 31 Jul 2010, Victor Julien wrote:
> Rmkml,
>
> You could try to set:
>
> detect-engine:
> - profile: low
>
> If that doesn't help, try setting it to "custom" and lower the values
> for the options below it.
>
> Also, the flow, defrag, engines prealloc's memory, try lowering this
> setting:
>
> flow:
> memcap: 33554432
> hash_size: 65536
> prealloc: 10000 # <--- this one
> emergency_recovery: 30
>
> defrag:
> max-frags: 65535
> prealloc: yes # <--- and this one
> timeout: 60
>
> Cheers,
> Victor
>
> rmkml at free.fr wrote:
>> Thx you Éric for comment,
>> But my max-pending-packets is 50 by default.
>> Regards
>> Rmkml
>>
>>
>> Selon Eric Leblond <eleblond at edenwall.com>:
>>
>>> Hi again,
>>>
>>> Le 30 juil. 2010 à 20:27, Eric Leblond <eleblond at edenwall.com> a écrit :
>>>
>>>> Hi,
>>>>
>>>> Le 30 juil. 2010 à 18:58, rmkml <rmkml at free.fr> a écrit :
>>>>
>>>>> Hi,
>>>>> Congratulations for Suricata v1.0.1!
>>>>> but this new release not fix my memory usage pb please.
>>>>> v101 - mem usage: 621M
>>>>> Im not continue my testing on your open source product because my linux
>>> kernel kill suricata process...
>>>> I ve myself experimented a heavy memory usage of suricata. It is linked
>>> with the max-pending-packets.
>>> Missing words here:
>>> This is a suricata.yaml variable and can thus be decreased. But I ve seen a
>>> huge performance improvement when increase it.
>>>
>>>> During init suricata preallocate this amount of packets.
>>>> But each Packet structure is of size 80384 and this can cause a huge memory
>>> usage.
>>>> BR
>>>>
>>>> Eric
>>>>
>>>>> Regards
>>>>> Rmkml
>>>>>
>>>>>
>>>>> On Mon, 26 Jul 2010, rmkml wrote:
>>>>>
>>>>>> It's ok, but with my commercial sig, suricata use 1.2G and killed by
>>> linux
>>>>>> kernel (on my personnal laptop).
>>>>>> Anyone test with vrt sigs please? (v2.8.5.3 or old)
>>>>>> Regards
>>>>>> Rmkml
>>>>>>
>>>>>>
>>>>>> On Mon, 26 Jul 2010, rmkml wrote:
>>>>>>
>>>>>>> Hi Victor,
>>>>>>> ok I have tested with theses suricata versions: (same conf, same pcap
>>> file
>>>>>>> is 27Mo)
>>>>>>> v100 - mem usage: 400M
>>>>>>> git13jul- mem usage: 400M
>>>>>>> git21jul- mem usage: 630M
>>>>>>> git25jul- mem usage: 649M
>>>>>>> All test with emerging all sigs daily
>>>>>>> (http://www.emergingthreats.net/rules/emerging-all.rules.zip)
>>>>>>> Anyone confirm increase 50% memory please?
>>>>>>> Regards
>>>>>>> Rmkml
>>>>>>>
>>>>>>>
>>>>>>> On Mon, 26 Jul 2010, Victor Julien wrote:
>>>>>>>
>>>>>>>> I think the increased mem usage is caused by fixing some accuracy
>>>>>>>> issues. As far as I can tell, it's not a bug of some kind.
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Victor
>>>>>>>>
>>>>>>>> rmkml wrote:
>>>>>>>>> Thx Anoop and Victor,
>>>>>>>>> ok crash/segfault fixed,
>>>>>>>>> but mem usage increase always exist on git
>>>>>>>>> c25921edf01c9f2d2e3c639037528ef5440c566e.
>>>>>>>>> Regards
>>>>>>>>> Rmkml
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sun, 25 Jul 2010, Victor Julien wrote:
>>>>>>>>>
>>>>>>>>>> Should be fixed in current master. Thanks guys!
>>>>>>>>>>
>>>>>>>>>> Anoop Saldanha wrote:
>>>>>>>>>>> Attached a new patch. Please don't apply the older one. Fixed a
>>> small
>>>>>>>>>>> typo in the unittest. It should pass now.
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jul 25, 2010 at 10:48 AM, Anoop Saldanha
>>> <poonaatsoc at gmail.com
>>>>>>>>>>> <mailto:poonaatsoc at gmail.com>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi rmkml. Can you please check it with this attached patch.
>>>>>>>>>>> Should
>>>>>>>>>>> fix it. Added an unittest to the patch as well.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sun, Jul 25, 2010 at 1:21 AM, <rmkml at free.fr
>>>>>>>>>>> <mailto:rmkml at free.fr>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Ok Im found my "crash" sig:
>>>>>>>>>>> alert udp any any -> any any (msg:"crash"; byte_test:4,>,2,0;
>>>>>>>>>>> byte_jump:1,0,relative; sid:11; )
>>>>>>>>>>> Regards
>>>>>>>>>>> Rmkml
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Selon rmkml <rmkml at free.fr <mailto:rmkml at free.fr>>:
>>>>>>>>>>>
>>>>>>>>>>>> thx for reply Victor,
>>>>>>>>>>>> no problemo:
>>>>>>>>>>>>
>>>>>>>>>>>> ...
>>>>>>>>>>>> [20560] 24/7/2010 -- 16:23:13 - (detect.c:302) <Error>
>>>>>>>>>>> (DetectLoadSigFile) --
>>>>>>>>>>>> [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Error parsing
>>>>>>>>>>> signature "alert tcp
>>>>>>>>>>>> $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-PHP
>>>>>>>>>>> shoutbox.php
>>>>>>>>>>>> access"; flow:to_server,established;
>>>>>>>>>>> uricontent:"/shoutbox.php";
>>>>>>>>>>>> reference:nessus,11668; classtype:web-application-activity;
>>>>>>>>>>> sid:2142;
>>>>>>>>>>>> rev:1;)" from file /home/test/snort/rules/web-php.rules at
>>>>>>>>>>> line 94
>>>>>>>>>>>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594) <Error>
>>>>>>>>>>>> (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>>>>>> preceding content or uricontent or pcre option
>>>>>>>>>>>> *** glibc detected ***
>>>>>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul201
>>>>>>>>>>>> 0/src/.libs/suricata: corrupted double-linked list:
>>>>>>>>>>> 0x0a51dea8 ***
>>>>>>>>>>>> ======= Backtrace: =========
>>>>>>>>>>>> /lib/libc.so.6[0xa9d06d]
>>>>>>>>>>>> ...
>>>>>>>>>>>>
>>>>>>>>>>>> Regards
>>>>>>>>>>>> Rmkml
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Can you share the signature this is happening with?
>>>>>>>>>>> Privately if you
>>>>>>>>>>>> prefer.
>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>> Victor
>>>>>>>>>>>>>
>>>>>>>>>>>>> rmkml wrote:
>>>>>>>>>>>>>> Hi Victor,
>>>>>>>>>>>>>> Thx for your work and your time on this project!
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I have "downloaded" (git clone) last Suricata version,
>>>>>>>>>>>>>> but I have a glibc error (git
>>>>>>>>>>> ead29dc6918f4524f1fae7e892d3f86dac117c0a):
>>>>>>>>>>>>>> ...
>>>>>>>>>>>>>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594)
>>>>>>>>>>> <Error>
>>>>>>>>>>>>>> (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>>>>>>>> preceding content or uricontent or pcre option
>>>>>>>>>>>>>> *** glibc detected ***
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata:
>>>>>>>>>>>>>> corrupted double-linked list: 0x0a51dea8 ***
>>>>>>>>>>>>>> ======= Backtrace: =========
>>>>>>>>>>>>>> /lib/libc.so.6[0xa9d06d]
>>>>>>>>>>>>>> /lib/libc.so.6[0xa9eb2b]
>>>>>>>>>>>>>> /lib/libc.so.6(cfree+0x90)[0xaa2430]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807b0dd]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c04a]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c1fb]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x806586e]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x8065d4b]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804bc70]
>>>>>>>>>>>>>> /lib/libc.so.6(__libc_start_main+0xe0)[0xa4cf70]
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804aa01]
>>>>>>>>>>>>>> ======= Memory map: ========
>>>>>>>>>>>>>> 0072c000-0073e000 r-xp 00000000 08:02 3700508
>>>>>>>>>>> /lib/libz.so.1.2.3
>>>>>>>>>>>>>> 0073e000-0073f000 rw-p 00011000 08:02 3700508
>>>>>>>>>>> /lib/libz.so.1.2.3
>>>>>>>>>>>>>> 00a18000-00a33000 r-xp 00000000 08:02 11817698
>>>>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>>>>>>>> 00a33000-00a34000 r--p 0001a000 08:02 11817698
>>>>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>>>>>>>> 00a34000-00a35000 rw-p 0001b000 08:02 11817698
>>>>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>>>>>>>> 00a37000-00b85000 r-xp 00000000 08:02 11817699
>>>>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>>>>>>>> 00b85000-00b87000 r--p 0014e000 08:02 11817699
>>>>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>>>>>>>> 00b87000-00b88000 rw-p 00150000 08:02 11817699
>>>>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>>>>>>>> 00b88000-00b8b000 rw-p 00000000 00:00 0
>>>>>>>>>>>>>> 00bbf000-00bd3000 r-xp 00000000 08:02 5434178
>>>>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>>>>>>>> 00bd3000-00bd4000 r--p 00013000 08:02 5434178
>>>>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>>>>>>>> 00bd4000-00bd5000 rw-p 00014000 08:02 5434178
>>>>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>>>>>>>> 00bd5000-00bd7000 rw-p 00000000 00:00 0
>>>>>>>>>>>>>> 00bee000-00c17000 r-xp 00000000 08:02 2078837
>>>>>>>>>>> /usr/lib/libpcap.so.0.9.7
>>>>>>>>>>>>>> 00c17000-00c19000 rw-p 00028000 08:02 2078837
>>>>>>>>>>> /usr/lib/libpcap.so.0.9.7
>>>>>>>>>>>>>> 00c58000-00c7f000 r-xp 00000000 08:02 5434342
>>>>>>>>>>> /lib/libpcre.so.0.0.1
>>>>>>>>>>>>>> 00c7f000-00c80000 rw-p 00026000 08:02 5434342
>>>>>>>>>>> /lib/libpcre.so.0.0.1
>>>>>>>>>>>>>> 05db4000-05dbf000 r-xp 00000000 08:02 5434249
>>>>>>>>>>>>>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>>>>>>>> 05dbf000-05dc0000 rw-p 0000a000 08:02 5434249
>>>>>>>>>>>>>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>>>>>>>> 08048000-08100000 r-xp 00000000 08:02 1244073
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>>>>>>>> 08100000-08101000 rw-p 000b8000 08:02 1244073
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>>>>>>>> 08101000-0a53d000 rw-p 00000000 00:00 0 [heap]
>>>>>>>>>>>>>> b7400000-b7421000 rw-p 00000000 00:00 0
>>>>>>>>>>>>>> b7421000-b7500000 ---p 00000000 00:00 0
>>>>>>>>>>>>>> b7594000-b771c000 rw-p 00000000 00:00 0
>>>>>>>>>>>>>> b771c000-b7737000 r-xp 00000000 08:02 11261710
>>>>>>>>>>>>>>
>>>>>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>>>>>>>> b7737000-b7738000 rw-p 0001a000 08:02 11261710
>>>>>>>>>>>>>>
>>>>>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>>>>>>>> b7748000-b7749000 rw-p 00000000 00:00 0
>>>>>>>>>>>>>> b7749000-b7758000 r-xp 00000000 08:02 654980
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>>>>>>>> b7758000-b7759000 rw-p 0000e000 08:02 654980
>>>>>>>>>>>>>>
>>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>>>>>>>> b7759000-b775a000 rw-p 00000000 00:00 0
>>>>>>>>>>>>>> b775a000-b775b000 r-xp 00000000 00:00 0 [vdso]
>>>>>>>>>>>>>> bf96c000-bf98d000 rw-p 00000000 00:00 0 [stack]
>>>>>>>>>>>>>> Abandon
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Regards
>>>>>>>>>>>>>> Rmkml
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> rmkml at free.fr <mailto:rmkml at free.fr> wrote:
>>>>>>>>>>>>>>>> I have new:
>>>>>>>>>>>>>>>> On git 21 jul, mem usage pb appear, but I have a small
>>>>>>>>>>> (revert)
>>>>>>>>>>>>>>>> change "resolv"
>>>>>>>>>>>>>>>> my pb, Move (back) this Line on if loop /* content */:
>>>>>>>>>>>>>>>> PatternMatchPreparePopulateMpm(de_ctx, sh);
>>>>>>>>>>>>>>>> #line 1081 in src/detect-engine-mpm.c
>>>>>>>>>>>>>>> Thanks Rmkml. At this point I don't think there is
>>>>>>>>>>> anything wrong in the
>>>>>>>>>>>>>>> code there. The changes were done to fix some accuracy
>>>>>>>>>>> issues we were
>>>>>>>>>>>>>>> seeing.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I did cleanup the code a bit in the latest git master. I
>>>>>>>>>>> don't expect
>>>>>>>>>>>>>>> anything to change, but maybe you can try anyway :)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>>> Victor
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>>> Victor Julien
>>>>>>>>>>>>>>> http://www.inliniac.net/
>>>>>>>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>> Victor Julien
>>>>>>>>>>>>> http://www.inliniac.net/
>>>>>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Oisf-devel mailing list
>>>>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>>>>> <mailto:Oisf-devel at openinfosecfoundation.org>
>>>>>>>>>>>
>>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Anoop Saldanha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Anoop Saldanha
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>> ------------------------------------------------------------------------
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Oisf-devel mailing list
>>>>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>> Victor Julien
>>>>>>>>>> http://www.inliniac.net/
>>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> ---------------------------------------------
>>>>>>>> Victor Julien
>>>>>>>> http://www.inliniac.net/
>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>> ---------------------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-devel mailing list
>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>
>>>>>>>
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>> _______________________________________________
>>>> Oisf-devel mailing list
>>>> Oisf-devel at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>
>>
>> _______________________________________________
>> Oisf-devel mailing list
>> Oisf-devel at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
More information about the Oisf-devel
mailing list