[Oisf-devel] Memory pb on Suricata git today
Victor Julien
victor at inliniac.net
Sat Jul 31 07:34:59 UTC 2010
Rmkml,
You could try to set:
detect-engine:
- profile: low
If that doesn't help, try setting it to "custom" and lower the values
for the options below it.
Also, the flow, defrag, engines prealloc's memory, try lowering this
setting:
flow:
memcap: 33554432
hash_size: 65536
prealloc: 10000 # <--- this one
emergency_recovery: 30
defrag:
max-frags: 65535
prealloc: yes # <--- and this one
timeout: 60
Cheers,
Victor
rmkml at free.fr wrote:
> Thx you Éric for comment,
> But my max-pending-packets is 50 by default.
> Regards
> Rmkml
>
>
> Selon Eric Leblond <eleblond at edenwall.com>:
>
>> Hi again,
>>
>> Le 30 juil. 2010 à 20:27, Eric Leblond <eleblond at edenwall.com> a écrit :
>>
>>> Hi,
>>>
>>> Le 30 juil. 2010 à 18:58, rmkml <rmkml at free.fr> a écrit :
>>>
>>>> Hi,
>>>> Congratulations for Suricata v1.0.1!
>>>> but this new release not fix my memory usage pb please.
>>>> v101 - mem usage: 621M
>>>> Im not continue my testing on your open source product because my linux
>> kernel kill suricata process...
>>> I ve myself experimented a heavy memory usage of suricata. It is linked
>> with the max-pending-packets.
>> Missing words here:
>> This is a suricata.yaml variable and can thus be decreased. But I ve seen a
>> huge performance improvement when increase it.
>>
>>> During init suricata preallocate this amount of packets.
>>> But each Packet structure is of size 80384 and this can cause a huge memory
>> usage.
>>> BR
>>>
>>> Eric
>>>
>>>> Regards
>>>> Rmkml
>>>>
>>>>
>>>> On Mon, 26 Jul 2010, rmkml wrote:
>>>>
>>>>> It's ok, but with my commercial sig, suricata use 1.2G and killed by
>> linux
>>>>> kernel (on my personnal laptop).
>>>>> Anyone test with vrt sigs please? (v2.8.5.3 or old)
>>>>> Regards
>>>>> Rmkml
>>>>>
>>>>>
>>>>> On Mon, 26 Jul 2010, rmkml wrote:
>>>>>
>>>>>> Hi Victor,
>>>>>> ok I have tested with theses suricata versions: (same conf, same pcap
>> file
>>>>>> is 27Mo)
>>>>>> v100 - mem usage: 400M
>>>>>> git13jul- mem usage: 400M
>>>>>> git21jul- mem usage: 630M
>>>>>> git25jul- mem usage: 649M
>>>>>> All test with emerging all sigs daily
>>>>>> (http://www.emergingthreats.net/rules/emerging-all.rules.zip)
>>>>>> Anyone confirm increase 50% memory please?
>>>>>> Regards
>>>>>> Rmkml
>>>>>>
>>>>>>
>>>>>> On Mon, 26 Jul 2010, Victor Julien wrote:
>>>>>>
>>>>>>> I think the increased mem usage is caused by fixing some accuracy
>>>>>>> issues. As far as I can tell, it's not a bug of some kind.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Victor
>>>>>>>
>>>>>>> rmkml wrote:
>>>>>>>> Thx Anoop and Victor,
>>>>>>>> ok crash/segfault fixed,
>>>>>>>> but mem usage increase always exist on git
>>>>>>>> c25921edf01c9f2d2e3c639037528ef5440c566e.
>>>>>>>> Regards
>>>>>>>> Rmkml
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, 25 Jul 2010, Victor Julien wrote:
>>>>>>>>
>>>>>>>>> Should be fixed in current master. Thanks guys!
>>>>>>>>>
>>>>>>>>> Anoop Saldanha wrote:
>>>>>>>>>> Attached a new patch. Please don't apply the older one. Fixed a
>> small
>>>>>>>>>> typo in the unittest. It should pass now.
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 25, 2010 at 10:48 AM, Anoop Saldanha
>> <poonaatsoc at gmail.com
>>>>>>>>>> <mailto:poonaatsoc at gmail.com>> wrote:
>>>>>>>>>>
>>>>>>>>>> Hi rmkml. Can you please check it with this attached patch.
>>>>>>>>>> Should
>>>>>>>>>> fix it. Added an unittest to the patch as well.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Sun, Jul 25, 2010 at 1:21 AM, <rmkml at free.fr
>>>>>>>>>> <mailto:rmkml at free.fr>> wrote:
>>>>>>>>>>
>>>>>>>>>> Ok Im found my "crash" sig:
>>>>>>>>>> alert udp any any -> any any (msg:"crash"; byte_test:4,>,2,0;
>>>>>>>>>> byte_jump:1,0,relative; sid:11; )
>>>>>>>>>> Regards
>>>>>>>>>> Rmkml
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Selon rmkml <rmkml at free.fr <mailto:rmkml at free.fr>>:
>>>>>>>>>>
>>>>>>>>>>> thx for reply Victor,
>>>>>>>>>>> no problemo:
>>>>>>>>>>>
>>>>>>>>>>> ...
>>>>>>>>>>> [20560] 24/7/2010 -- 16:23:13 - (detect.c:302) <Error>
>>>>>>>>>> (DetectLoadSigFile) --
>>>>>>>>>>> [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Error parsing
>>>>>>>>>> signature "alert tcp
>>>>>>>>>>> $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-PHP
>>>>>>>>>> shoutbox.php
>>>>>>>>>>> access"; flow:to_server,established;
>>>>>>>>>> uricontent:"/shoutbox.php";
>>>>>>>>>>> reference:nessus,11668; classtype:web-application-activity;
>>>>>>>>>> sid:2142;
>>>>>>>>>>> rev:1;)" from file /home/test/snort/rules/web-php.rules at
>>>>>>>>>> line 94
>>>>>>>>>>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594) <Error>
>>>>>>>>>>> (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>>>>> preceding content or uricontent or pcre option
>>>>>>>>>>> *** glibc detected ***
>>>>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul201
>>>>>>>>>>> 0/src/.libs/suricata: corrupted double-linked list:
>>>>>>>>>> 0x0a51dea8 ***
>>>>>>>>>>> ======= Backtrace: =========
>>>>>>>>>>> /lib/libc.so.6[0xa9d06d]
>>>>>>>>>>> ...
>>>>>>>>>>>
>>>>>>>>>>> Regards
>>>>>>>>>>> Rmkml
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you share the signature this is happening with?
>>>>>>>>>> Privately if you
>>>>>>>>>>> prefer.
>>>>>>>>>>>> Cheers,
>>>>>>>>>>>> Victor
>>>>>>>>>>>>
>>>>>>>>>>>> rmkml wrote:
>>>>>>>>>>>>> Hi Victor,
>>>>>>>>>>>>> Thx for your work and your time on this project!
>>>>>>>>>>>>>
>>>>>>>>>>>>> I have "downloaded" (git clone) last Suricata version,
>>>>>>>>>>>>> but I have a glibc error (git
>>>>>>>>>> ead29dc6918f4524f1fae7e892d3f86dac117c0a):
>>>>>>>>>>>>> ...
>>>>>>>>>>>>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594)
>>>>>>>>>> <Error>
>>>>>>>>>>>>> (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>>>>>>> preceding content or uricontent or pcre option
>>>>>>>>>>>>> *** glibc detected ***
>>>>>>>>>>>>>
>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata:
>>>>>>>>>>>>> corrupted double-linked list: 0x0a51dea8 ***
>>>>>>>>>>>>> ======= Backtrace: =========
>>>>>>>>>>>>> /lib/libc.so.6[0xa9d06d]
>>>>>>>>>>>>> /lib/libc.so.6[0xa9eb2b]
>>>>>>>>>>>>> /lib/libc.so.6(cfree+0x90)[0xaa2430]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807b0dd]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c04a]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c1fb]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x806586e]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x8065d4b]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804bc70]
>>>>>>>>>>>>> /lib/libc.so.6(__libc_start_main+0xe0)[0xa4cf70]
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804aa01]
>>>>>>>>>>>>> ======= Memory map: ========
>>>>>>>>>>>>> 0072c000-0073e000 r-xp 00000000 08:02 3700508
>>>>>>>>>> /lib/libz.so.1.2.3
>>>>>>>>>>>>> 0073e000-0073f000 rw-p 00011000 08:02 3700508
>>>>>>>>>> /lib/libz.so.1.2.3
>>>>>>>>>>>>> 00a18000-00a33000 r-xp 00000000 08:02 11817698
>>>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>>>>>>> 00a33000-00a34000 r--p 0001a000 08:02 11817698
>>>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>>>>>>> 00a34000-00a35000 rw-p 0001b000 08:02 11817698
>>>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>>>>>>> 00a37000-00b85000 r-xp 00000000 08:02 11817699
>>>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>>>>>>> 00b85000-00b87000 r--p 0014e000 08:02 11817699
>>>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>>>>>>> 00b87000-00b88000 rw-p 00150000 08:02 11817699
>>>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>>>>>>> 00b88000-00b8b000 rw-p 00000000 00:00 0
>>>>>>>>>>>>> 00bbf000-00bd3000 r-xp 00000000 08:02 5434178
>>>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>>>>>>> 00bd3000-00bd4000 r--p 00013000 08:02 5434178
>>>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>>>>>>> 00bd4000-00bd5000 rw-p 00014000 08:02 5434178
>>>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>>>>>>> 00bd5000-00bd7000 rw-p 00000000 00:00 0
>>>>>>>>>>>>> 00bee000-00c17000 r-xp 00000000 08:02 2078837
>>>>>>>>>> /usr/lib/libpcap.so.0.9.7
>>>>>>>>>>>>> 00c17000-00c19000 rw-p 00028000 08:02 2078837
>>>>>>>>>> /usr/lib/libpcap.so.0.9.7
>>>>>>>>>>>>> 00c58000-00c7f000 r-xp 00000000 08:02 5434342
>>>>>>>>>> /lib/libpcre.so.0.0.1
>>>>>>>>>>>>> 00c7f000-00c80000 rw-p 00026000 08:02 5434342
>>>>>>>>>> /lib/libpcre.so.0.0.1
>>>>>>>>>>>>> 05db4000-05dbf000 r-xp 00000000 08:02 5434249
>>>>>>>>>>>>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>>>>>>> 05dbf000-05dc0000 rw-p 0000a000 08:02 5434249
>>>>>>>>>>>>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>>>>>>> 08048000-08100000 r-xp 00000000 08:02 1244073
>>>>>>>>>>>>>
>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>>>>>>> 08100000-08101000 rw-p 000b8000 08:02 1244073
>>>>>>>>>>>>>
>>>>>>>>>>
>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>>>>>>> 08101000-0a53d000 rw-p 00000000 00:00 0 [heap]
>>>>>>>>>>>>> b7400000-b7421000 rw-p 00000000 00:00 0
>>>>>>>>>>>>> b7421000-b7500000 ---p 00000000 00:00 0
>>>>>>>>>>>>> b7594000-b771c000 rw-p 00000000 00:00 0
>>>>>>>>>>>>> b771c000-b7737000 r-xp 00000000 08:02 11261710
>>>>>>>>>>>>>
>>>>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>>>>>>> b7737000-b7738000 rw-p 0001a000 08:02 11261710
>>>>>>>>>>>>>
>>>>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>>>>>>> b7748000-b7749000 rw-p 00000000 00:00 0
>>>>>>>>>>>>> b7749000-b7758000 r-xp 00000000 08:02 654980
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>>>>>>> b7758000-b7759000 rw-p 0000e000 08:02 654980
>>>>>>>>>>>>>
>>>>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>>>>>>> b7759000-b775a000 rw-p 00000000 00:00 0
>>>>>>>>>>>>> b775a000-b775b000 r-xp 00000000 00:00 0 [vdso]
>>>>>>>>>>>>> bf96c000-bf98d000 rw-p 00000000 00:00 0 [stack]
>>>>>>>>>>>>> Abandon
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards
>>>>>>>>>>>>> Rmkml
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> rmkml at free.fr <mailto:rmkml at free.fr> wrote:
>>>>>>>>>>>>>>> I have new:
>>>>>>>>>>>>>>> On git 21 jul, mem usage pb appear, but I have a small
>>>>>>>>>> (revert)
>>>>>>>>>>>>>>> change "resolv"
>>>>>>>>>>>>>>> my pb, Move (back) this Line on if loop /* content */:
>>>>>>>>>>>>>>> PatternMatchPreparePopulateMpm(de_ctx, sh);
>>>>>>>>>>>>>>> #line 1081 in src/detect-engine-mpm.c
>>>>>>>>>>>>>> Thanks Rmkml. At this point I don't think there is
>>>>>>>>>> anything wrong in the
>>>>>>>>>>>>>> code there. The changes were done to fix some accuracy
>>>>>>>>>> issues we were
>>>>>>>>>>>>>> seeing.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I did cleanup the code a bit in the latest git master. I
>>>>>>>>>> don't expect
>>>>>>>>>>>>>> anything to change, but maybe you can try anyway :)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Cheers,
>>>>>>>>>>>>>> Victor
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>> Victor Julien
>>>>>>>>>>>>>> http://www.inliniac.net/
>>>>>>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>> Victor Julien
>>>>>>>>>>>> http://www.inliniac.net/
>>>>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>>>>>> ---------------------------------------------
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Oisf-devel mailing list
>>>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>>>> <mailto:Oisf-devel at openinfosecfoundation.org>
>>>>>>>>>>
>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Anoop Saldanha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Anoop Saldanha
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>> ------------------------------------------------------------------------
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Oisf-devel mailing list
>>>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ---------------------------------------------
>>>>>>>>> Victor Julien
>>>>>>>>> http://www.inliniac.net/
>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>>> ---------------------------------------------
>>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ---------------------------------------------
>>>>>>> Victor Julien
>>>>>>> http://www.inliniac.net/
>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Oisf-devel mailing list
>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>
>>>>>>
>>>> _______________________________________________
>>>> Oisf-devel mailing list
>>>> Oisf-devel at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>> _______________________________________________
>>> Oisf-devel mailing list
>>> Oisf-devel at openinfosecfoundation.org
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
More information about the Oisf-devel
mailing list