[Oisf-devel] Memory pb on Suricata git today
rmkml
rmkml at free.fr
Sat Jul 31 17:29:01 UTC 2010
Hi Victor,
I load always (for testing) emerging all: 8380 rules.
For information, my personnal ruleset have 7595 (less than emerging but I
have memory usage pb)
Im try your recommandation...
Thx
Rmkml
On Sat, 31 Jul 2010, Victor Julien wrote:
> rmkml wrote:
>> Hi,
>> Congratulations for Suricata v1.0.1!
>> but this new release not fix my memory usage pb please.
>> v101 - mem usage: 621M
>
> How many signatures did you load?
>
> Cheers,
> Victor
>
>
>> Im not continue my testing on your open source product because my linux kernel kill suricata process...
>> Regards
>> Rmkml
>>
>>
>> On Mon, 26 Jul 2010, rmkml wrote:
>>
>>> It's ok, but with my commercial sig, suricata use 1.2G and killed by linux
>>> kernel (on my personnal laptop).
>>> Anyone test with vrt sigs please? (v2.8.5.3 or old)
>>> Regards
>>> Rmkml
>>>
>>>
>>> On Mon, 26 Jul 2010, rmkml wrote:
>>>
>>>> Hi Victor,
>>>> ok I have tested with theses suricata versions: (same conf, same pcap file
>>>> is 27Mo)
>>>> v100 - mem usage: 400M
>>>> git13jul- mem usage: 400M
>>>> git21jul- mem usage: 630M
>>>> git25jul- mem usage: 649M
>>>> All test with emerging all sigs daily
>>>> (http://www.emergingthreats.net/rules/emerging-all.rules.zip)
>>>> Anyone confirm increase 50% memory please?
>>>> Regards
>>>> Rmkml
>>>>
>>>>
>>>> On Mon, 26 Jul 2010, Victor Julien wrote:
>>>>
>>>>> I think the increased mem usage is caused by fixing some accuracy
>>>>> issues. As far as I can tell, it's not a bug of some kind.
>>>>>
>>>>> Cheers,
>>>>> Victor
>>>>>
>>>>> rmkml wrote:
>>>>>> Thx Anoop and Victor,
>>>>>> ok crash/segfault fixed,
>>>>>> but mem usage increase always exist on git
>>>>>> c25921edf01c9f2d2e3c639037528ef5440c566e.
>>>>>> Regards
>>>>>> Rmkml
>>>>>>
>>>>>>
>>>>>> On Sun, 25 Jul 2010, Victor Julien wrote:
>>>>>>
>>>>>>> Should be fixed in current master. Thanks guys!
>>>>>>>
>>>>>>> Anoop Saldanha wrote:
>>>>>>>> Attached a new patch. Please don't apply the older one. Fixed a small
>>>>>>>> typo in the unittest. It should pass now.
>>>>>>>>
>>>>>>>> On Sun, Jul 25, 2010 at 10:48 AM, Anoop Saldanha <poonaatsoc at gmail.com
>>>>>>>> <mailto:poonaatsoc at gmail.com>> wrote:
>>>>>>>>
>>>>>>>> Hi rmkml. Can you please check it with this attached patch.
>>>>>>>> Should
>>>>>>>> fix it. Added an unittest to the patch as well.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sun, Jul 25, 2010 at 1:21 AM, <rmkml at free.fr
>>>>>>>> <mailto:rmkml at free.fr>> wrote:
>>>>>>>>
>>>>>>>> Ok Im found my "crash" sig:
>>>>>>>> alert udp any any -> any any (msg:"crash"; byte_test:4,>,2,0;
>>>>>>>> byte_jump:1,0,relative; sid:11; )
>>>>>>>> Regards
>>>>>>>> Rmkml
>>>>>>>>
>>>>>>>>
>>>>>>>> Selon rmkml <rmkml at free.fr <mailto:rmkml at free.fr>>:
>>>>>>>>
>>>>>>>> > thx for reply Victor,
>>>>>>>> > no problemo:
>>>>>>>> >
>>>>>>>> > ...
>>>>>>>> > [20560] 24/7/2010 -- 16:23:13 - (detect.c:302) <Error>
>>>>>>>> (DetectLoadSigFile) --
>>>>>>>> > [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Error parsing
>>>>>>>> signature "alert tcp
>>>>>>>> > $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-PHP
>>>>>>>> shoutbox.php
>>>>>>>> > access"; flow:to_server,established;
>>>>>>>> uricontent:"/shoutbox.php";
>>>>>>>> > reference:nessus,11668; classtype:web-application-activity;
>>>>>>>> sid:2142;
>>>>>>>> > rev:1;)" from file /home/test/snort/rules/web-php.rules at
>>>>>>>> line 94
>>>>>>>> > [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594) <Error>
>>>>>>>> > (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>> > preceding content or uricontent or pcre option
>>>>>>>> > *** glibc detected ***
>>>>>>>> > /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul201
>>>>>>>> > 0/src/.libs/suricata: corrupted double-linked list:
>>>>>>>> 0x0a51dea8 ***
>>>>>>>> > ======= Backtrace: =========
>>>>>>>> > /lib/libc.so.6[0xa9d06d]
>>>>>>>> > ...
>>>>>>>> >
>>>>>>>> > Regards
>>>>>>>> > Rmkml
>>>>>>>> >
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>> >
>>>>>>>> >> Can you share the signature this is happening with?
>>>>>>>> Privately if you
>>>>>>>> > prefer.
>>>>>>>> >>
>>>>>>>> >> Cheers,
>>>>>>>> >> Victor
>>>>>>>> >>
>>>>>>>> >> rmkml wrote:
>>>>>>>> >>> Hi Victor,
>>>>>>>> >>> Thx for your work and your time on this project!
>>>>>>>> >>>
>>>>>>>> >>> I have "downloaded" (git clone) last Suricata version,
>>>>>>>> >>> but I have a glibc error (git
>>>>>>>> ead29dc6918f4524f1fae7e892d3f86dac117c0a):
>>>>>>>> >>> ...
>>>>>>>> >>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594)
>>>>>>>> <Error>
>>>>>>>> >>> (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>> >>> preceding content or uricontent or pcre option
>>>>>>>> >>> *** glibc detected ***
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata:
>>>>>>>>
>>>>>>>> >>> corrupted double-linked list: 0x0a51dea8 ***
>>>>>>>> >>> ======= Backtrace: =========
>>>>>>>> >>> /lib/libc.so.6[0xa9d06d]
>>>>>>>> >>> /lib/libc.so.6[0xa9eb2b]
>>>>>>>> >>> /lib/libc.so.6(cfree+0x90)[0xaa2430]
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807b0dd]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c04a]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c1fb]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x806586e]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x8065d4b]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804bc70]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>> /lib/libc.so.6(__libc_start_main+0xe0)[0xa4cf70]
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804aa01]
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>> ======= Memory map: ========
>>>>>>>> >>> 0072c000-0073e000 r-xp 00000000 08:02 3700508
>>>>>>>> /lib/libz.so.1.2.3
>>>>>>>> >>> 0073e000-0073f000 rw-p 00011000 08:02 3700508
>>>>>>>> /lib/libz.so.1.2.3
>>>>>>>> >>> 00a18000-00a33000 r-xp 00000000 08:02 11817698
>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>> >>> 00a33000-00a34000 r--p 0001a000 08:02 11817698
>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>> >>> 00a34000-00a35000 rw-p 0001b000 08:02 11817698
>>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>> >>> 00a37000-00b85000 r-xp 00000000 08:02 11817699
>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>> >>> 00b85000-00b87000 r--p 0014e000 08:02 11817699
>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>> >>> 00b87000-00b88000 rw-p 00150000 08:02 11817699
>>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>> >>> 00b88000-00b8b000 rw-p 00000000 00:00 0
>>>>>>>> >>> 00bbf000-00bd3000 r-xp 00000000 08:02 5434178
>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>> >>> 00bd3000-00bd4000 r--p 00013000 08:02 5434178
>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>> >>> 00bd4000-00bd5000 rw-p 00014000 08:02 5434178
>>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>> >>> 00bd5000-00bd7000 rw-p 00000000 00:00 0
>>>>>>>> >>> 00bee000-00c17000 r-xp 00000000 08:02 2078837
>>>>>>>> /usr/lib/libpcap.so.0.9.7
>>>>>>>> >>> 00c17000-00c19000 rw-p 00028000 08:02 2078837
>>>>>>>> /usr/lib/libpcap.so.0.9.7
>>>>>>>> >>> 00c58000-00c7f000 r-xp 00000000 08:02 5434342
>>>>>>>> /lib/libpcre.so.0.0.1
>>>>>>>> >>> 00c7f000-00c80000 rw-p 00026000 08:02 5434342
>>>>>>>> /lib/libpcre.so.0.0.1
>>>>>>>> >>> 05db4000-05dbf000 r-xp 00000000 08:02 5434249
>>>>>>>> >>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>> >>> 05dbf000-05dc0000 rw-p 0000a000 08:02 5434249
>>>>>>>> >>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>> >>> 08048000-08100000 r-xp 00000000 08:02 1244073
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>> 08100000-08101000 rw-p 000b8000 08:02 1244073
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>> 08101000-0a53d000 rw-p 00000000 00:00 0 [heap]
>>>>>>>> >>> b7400000-b7421000 rw-p 00000000 00:00 0
>>>>>>>> >>> b7421000-b7500000 ---p 00000000 00:00 0
>>>>>>>> >>> b7594000-b771c000 rw-p 00000000 00:00 0
>>>>>>>> >>> b771c000-b7737000 r-xp 00000000 08:02 11261710
>>>>>>>> >>>
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>> >>> b7737000-b7738000 rw-p 0001a000 08:02 11261710
>>>>>>>> >>>
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>> >>> b7748000-b7749000 rw-p 00000000 00:00 0
>>>>>>>> >>> b7749000-b7758000 r-xp 00000000 08:02 654980
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>> b7758000-b7759000 rw-p 0000e000 08:02 654980
>>>>>>>> >>>
>>>>>>>> >
>>>>>>>>
>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>>
>>>>>>>> >>>
>>>>>>>> >>> b7759000-b775a000 rw-p 00000000 00:00 0
>>>>>>>> >>> b775a000-b775b000 r-xp 00000000 00:00 0 [vdso]
>>>>>>>> >>> bf96c000-bf98d000 rw-p 00000000 00:00 0 [stack]
>>>>>>>> >>> Abandon
>>>>>>>> >>>
>>>>>>>> >>> Regards
>>>>>>>> >>> Rmkml
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>>
>>>>>>>> >>> On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>> >>>
>>>>>>>> >>>> rmkml at free.fr <mailto:rmkml at free.fr> wrote:
>>>>>>>> >>>>> I have new:
>>>>>>>> >>>>> On git 21 jul, mem usage pb appear, but I have a small
>>>>>>>> (revert)
>>>>>>>> >>>>> change "resolv"
>>>>>>>> >>>>> my pb, Move (back) this Line on if loop /* content */:
>>>>>>>> >>>>> PatternMatchPreparePopulateMpm(de_ctx, sh);
>>>>>>>> >>>>> #line 1081 in src/detect-engine-mpm.c
>>>>>>>> >>>>
>>>>>>>> >>>> Thanks Rmkml. At this point I don't think there is
>>>>>>>> anything wrong in the
>>>>>>>> >>>> code there. The changes were done to fix some accuracy
>>>>>>>> issues we were
>>>>>>>> >>>> seeing.
>>>>>>>> >>>>
>>>>>>>> >>>> I did cleanup the code a bit in the latest git master. I
>>>>>>>> don't expect
>>>>>>>> >>>> anything to change, but maybe you can try anyway :)
>>>>>>>> >>>>
>>>>>>>> >>>> Cheers,
>>>>>>>> >>>> Victor
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>>> --
>>>>>>>> >>>> ---------------------------------------------
>>>>>>>> >>>> Victor Julien
>>>>>>>> >>>> http://www.inliniac.net/
>>>>>>>> >>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>> >>>> ---------------------------------------------
>>>>>>>> >>>>
>>>>>>>> >>>>
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >> --
>>>>>>>> >> ---------------------------------------------
>>>>>>>> >> Victor Julien
>>>>>>>> >> http://www.inliniac.net/
>>>>>>>> >> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>> >> ---------------------------------------------
>>>>>>>> >>
>>>>>>>> >>
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-devel mailing list
>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>> <mailto:Oisf-devel at openinfosecfoundation.org>
>>>>>>>>
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Anoop Saldanha
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Anoop Saldanha
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------------
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-devel mailing list
>>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>
>>>>>>> --
>>>>>>> ---------------------------------------------
>>>>>>> Victor Julien
>>>>>>> http://www.inliniac.net/
>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>> ---------------------------------------------
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> ---------------------------------------------
>>>>> Victor Julien
>>>>> http://www.inliniac.net/
>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>> ---------------------------------------------
>>>>>
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>
>>>>
>> _______________________________________________
>> Oisf-devel mailing list
>> Oisf-devel at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
More information about the Oisf-devel
mailing list