[Oisf-devel] Memory pb on Suricata git today

Victor Julien victor at inliniac.net
Sat Jul 31 07:36:01 UTC 2010


rmkml wrote:
> Hi,
> Congratulations for Suricata v1.0.1!
> but this new release not fix my memory usage pb please.
>   v101    - mem usage: 621M

How many signatures did you load?

Cheers,
Victor


> Im not continue my testing on your open source product because my linux kernel kill suricata process...
> Regards
> Rmkml
> 
> 
> On Mon, 26 Jul 2010, rmkml wrote:
> 
>> It's ok, but with my commercial sig, suricata use 1.2G and killed by linux 
>> kernel (on my personnal laptop).
>> Anyone test with vrt sigs please? (v2.8.5.3 or old)
>> Regards
>> Rmkml
>>
>>
>> On Mon, 26 Jul 2010, rmkml wrote:
>>
>>> Hi Victor,
>>> ok I have tested with theses suricata versions: (same conf, same pcap file 
>>> is 27Mo)
>>> v100    - mem usage: 400M
>>> git13jul- mem usage: 400M
>>> git21jul- mem usage: 630M
>>> git25jul- mem usage: 649M
>>> All test with emerging all sigs daily 
>>> (http://www.emergingthreats.net/rules/emerging-all.rules.zip)
>>> Anyone confirm increase 50% memory please?
>>> Regards
>>> Rmkml
>>>
>>>
>>> On Mon, 26 Jul 2010, Victor Julien wrote:
>>>
>>>> I think the increased mem usage is caused by fixing some accuracy
>>>> issues. As far as I can tell, it's not a bug of some kind.
>>>>
>>>> Cheers,
>>>> Victor
>>>>
>>>> rmkml wrote:
>>>>> Thx Anoop and Victor,
>>>>> ok crash/segfault fixed,
>>>>> but mem usage increase always exist on git
>>>>> c25921edf01c9f2d2e3c639037528ef5440c566e.
>>>>> Regards
>>>>> Rmkml
>>>>>
>>>>>
>>>>> On Sun, 25 Jul 2010, Victor Julien wrote:
>>>>>
>>>>>> Should be fixed in current master. Thanks guys!
>>>>>>
>>>>>> Anoop Saldanha wrote:
>>>>>>> Attached a new patch.  Please don't apply the older one.  Fixed a small
>>>>>>> typo in the unittest.  It should pass now.
>>>>>>>
>>>>>>> On Sun, Jul 25, 2010 at 10:48 AM, Anoop Saldanha <poonaatsoc at gmail.com
>>>>>>> <mailto:poonaatsoc at gmail.com>> wrote:
>>>>>>>
>>>>>>>     Hi rmkml.  Can you please check it with this attached patch. 
>>>>>>> Should
>>>>>>>     fix it.  Added an unittest to the patch as well.
>>>>>>>
>>>>>>>
>>>>>>>     On Sun, Jul 25, 2010 at 1:21 AM, <rmkml at free.fr
>>>>>>>     <mailto:rmkml at free.fr>> wrote:
>>>>>>>
>>>>>>>         Ok Im found my "crash" sig:
>>>>>>>         alert udp any any -> any any (msg:"crash"; byte_test:4,>,2,0;
>>>>>>>         byte_jump:1,0,relative; sid:11; )
>>>>>>>         Regards
>>>>>>>         Rmkml
>>>>>>>
>>>>>>>
>>>>>>>         Selon rmkml <rmkml at free.fr <mailto:rmkml at free.fr>>:
>>>>>>>
>>>>>>>       > thx for reply Victor,
>>>>>>>       > no problemo:
>>>>>>>       >
>>>>>>>       > ...
>>>>>>>       > [20560] 24/7/2010 -- 16:23:13 - (detect.c:302) <Error>
>>>>>>>         (DetectLoadSigFile) --
>>>>>>>       > [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Error parsing
>>>>>>>         signature "alert tcp
>>>>>>>       > $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-PHP
>>>>>>>         shoutbox.php
>>>>>>>       > access"; flow:to_server,established;
>>>>>>> uricontent:"/shoutbox.php";
>>>>>>>       > reference:nessus,11668; classtype:web-application-activity;
>>>>>>>         sid:2142;
>>>>>>>       > rev:1;)" from file /home/test/snort/rules/web-php.rules at
>>>>>>> line 94
>>>>>>>       > [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594) <Error>
>>>>>>>       > (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>         SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>       > preceding content or uricontent or pcre option
>>>>>>>       > *** glibc detected ***
>>>>>>>       > /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul201
>>>>>>>       > 0/src/.libs/suricata: corrupted double-linked list:
>>>>>>> 0x0a51dea8 ***
>>>>>>>       > ======= Backtrace: =========
>>>>>>>       > /lib/libc.so.6[0xa9d06d]
>>>>>>>       > ...
>>>>>>>       >
>>>>>>>       > Regards
>>>>>>>       > Rmkml
>>>>>>>       >
>>>>>>>       >
>>>>>>>       >
>>>>>>>       > On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>       >
>>>>>>>       >> Can you share the signature this is happening with?
>>>>>>>         Privately if you
>>>>>>>       > prefer.
>>>>>>>       >>
>>>>>>>       >> Cheers,
>>>>>>>       >> Victor
>>>>>>>       >>
>>>>>>>       >> rmkml wrote:
>>>>>>>       >>> Hi Victor,
>>>>>>>       >>> Thx for your work and your time on this project!
>>>>>>>       >>>
>>>>>>>       >>> I have "downloaded" (git clone) last Suricata version,
>>>>>>>       >>> but I have a glibc error (git
>>>>>>>         ead29dc6918f4524f1fae7e892d3f86dac117c0a):
>>>>>>>       >>> ...
>>>>>>>       >>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594)
>>>>>>> <Error>
>>>>>>>       >>> (DetectBytejumpSetup) -- [ERRCODE:
>>>>>>>         SC_ERR_INVALID_SIGNATURE(39)] - No
>>>>>>>       >>> preceding content or uricontent or pcre option
>>>>>>>       >>> *** glibc detected ***
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata:
>>>>>>>
>>>>>>>       >>> corrupted double-linked list: 0x0a51dea8 ***
>>>>>>>       >>> ======= Backtrace: =========
>>>>>>>       >>> /lib/libc.so.6[0xa9d06d]
>>>>>>>       >>> /lib/libc.so.6[0xa9eb2b]
>>>>>>>       >>> /lib/libc.so.6(cfree+0x90)[0xaa2430]
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807b0dd]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c04a]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c1fb]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x806586e]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x8065d4b]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804bc70]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>> /lib/libc.so.6(__libc_start_main+0xe0)[0xa4cf70]
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804aa01]
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>> ======= Memory map: ========
>>>>>>>       >>> 0072c000-0073e000 r-xp 00000000 08:02 3700508
>>>>>>>          /lib/libz.so.1.2.3
>>>>>>>       >>> 0073e000-0073f000 rw-p 00011000 08:02 3700508
>>>>>>>          /lib/libz.so.1.2.3
>>>>>>>       >>> 00a18000-00a33000 r-xp 00000000 08:02 11817698
>>>>>>>         /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>       >>> 00a33000-00a34000 r--p 0001a000 08:02 11817698
>>>>>>>         /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>       >>> 00a34000-00a35000 rw-p 0001b000 08:02 11817698
>>>>>>>         /lib/ld-2.6.so <http://ld-2.6.so>
>>>>>>>       >>> 00a37000-00b85000 r-xp 00000000 08:02 11817699
>>>>>>>         /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>       >>> 00b85000-00b87000 r--p 0014e000 08:02 11817699
>>>>>>>         /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>       >>> 00b87000-00b88000 rw-p 00150000 08:02 11817699
>>>>>>>         /lib/libc-2.6.so <http://libc-2.6.so>
>>>>>>>       >>> 00b88000-00b8b000 rw-p 00000000 00:00 0
>>>>>>>       >>> 00bbf000-00bd3000 r-xp 00000000 08:02 5434178
>>>>>>>          /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>       >>> 00bd3000-00bd4000 r--p 00013000 08:02 5434178
>>>>>>>          /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>       >>> 00bd4000-00bd5000 rw-p 00014000 08:02 5434178
>>>>>>>          /lib/libpthread-2.6.so <http://libpthread-2.6.so>
>>>>>>>       >>> 00bd5000-00bd7000 rw-p 00000000 00:00 0
>>>>>>>       >>> 00bee000-00c17000 r-xp 00000000 08:02 2078837
>>>>>>>          /usr/lib/libpcap.so.0.9.7
>>>>>>>       >>> 00c17000-00c19000 rw-p 00028000 08:02 2078837
>>>>>>>          /usr/lib/libpcap.so.0.9.7
>>>>>>>       >>> 00c58000-00c7f000 r-xp 00000000 08:02 5434342
>>>>>>>          /lib/libpcre.so.0.0.1
>>>>>>>       >>> 00c7f000-00c80000 rw-p 00026000 08:02 5434342
>>>>>>>          /lib/libpcre.so.0.0.1
>>>>>>>       >>> 05db4000-05dbf000 r-xp 00000000 08:02 5434249
>>>>>>>       >>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>       >>> 05dbf000-05dc0000 rw-p 0000a000 08:02 5434249
>>>>>>>       >>> /lib/libgcc_s-4.1.2-20070925.so.1
>>>>>>>       >>> 08048000-08100000 r-xp 00000000 08:02 1244073
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>> 08100000-08101000 rw-p 000b8000 08:02 1244073
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>> 08101000-0a53d000 rw-p 00000000 00:00 0          [heap]
>>>>>>>       >>> b7400000-b7421000 rw-p 00000000 00:00 0
>>>>>>>       >>> b7421000-b7500000 ---p 00000000 00:00 0
>>>>>>>       >>> b7594000-b771c000 rw-p 00000000 00:00 0
>>>>>>>       >>> b771c000-b7737000 r-xp 00000000 08:02 11261710
>>>>>>>       >>>
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>       >>> b7737000-b7738000 rw-p 0001a000 08:02 11261710
>>>>>>>       >>>
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
>>>>>>>       >>> b7748000-b7749000 rw-p 00000000 00:00 0
>>>>>>>       >>> b7749000-b7758000 r-xp 00000000 08:02 654980
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>> b7758000-b7759000 rw-p 0000e000 08:02 654980
>>>>>>>       >>>
>>>>>>>       >
>>>>>>>
>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
>>>>>>>
>>>>>>>       >>>
>>>>>>>       >>> b7759000-b775a000 rw-p 00000000 00:00 0
>>>>>>>       >>> b775a000-b775b000 r-xp 00000000 00:00 0          [vdso]
>>>>>>>       >>> bf96c000-bf98d000 rw-p 00000000 00:00 0          [stack]
>>>>>>>       >>> Abandon
>>>>>>>       >>>
>>>>>>>       >>> Regards
>>>>>>>       >>> Rmkml
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >>>
>>>>>>>       >>> On Sat, 24 Jul 2010, Victor Julien wrote:
>>>>>>>       >>>
>>>>>>>       >>>> rmkml at free.fr <mailto:rmkml at free.fr> wrote:
>>>>>>>       >>>>> I have new:
>>>>>>>       >>>>> On git 21 jul, mem usage pb appear, but I have a small
>>>>>>>         (revert)
>>>>>>>       >>>>> change "resolv"
>>>>>>>       >>>>> my pb, Move (back) this Line on if loop /* content */:
>>>>>>>       >>>>>  PatternMatchPreparePopulateMpm(de_ctx, sh);
>>>>>>>       >>>>> #line 1081 in src/detect-engine-mpm.c
>>>>>>>       >>>>
>>>>>>>       >>>> Thanks Rmkml. At this point I don't think there is
>>>>>>>         anything wrong in the
>>>>>>>       >>>> code there. The changes were done to fix some accuracy
>>>>>>>         issues we were
>>>>>>>       >>>> seeing.
>>>>>>>       >>>>
>>>>>>>       >>>> I did cleanup the code a bit in the latest git master. I
>>>>>>>         don't expect
>>>>>>>       >>>> anything to change, but maybe you can try anyway :)
>>>>>>>       >>>>
>>>>>>>       >>>> Cheers,
>>>>>>>       >>>> Victor
>>>>>>>       >>>>
>>>>>>>       >>>>
>>>>>>>       >>>> --
>>>>>>>       >>>> ---------------------------------------------
>>>>>>>       >>>> Victor Julien
>>>>>>>       >>>> http://www.inliniac.net/
>>>>>>>       >>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>       >>>> ---------------------------------------------
>>>>>>>       >>>>
>>>>>>>       >>>>
>>>>>>>       >>
>>>>>>>       >>
>>>>>>>       >> --
>>>>>>>       >> ---------------------------------------------
>>>>>>>       >> Victor Julien
>>>>>>>       >> http://www.inliniac.net/
>>>>>>>       >> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>>>       >> ---------------------------------------------
>>>>>>>       >>
>>>>>>>       >>
>>>>>>>       >
>>>>>>>       >
>>>>>>>
>>>>>>>
>>>>>>>         _______________________________________________
>>>>>>>         Oisf-devel mailing list
>>>>>>>         Oisf-devel at openinfosecfoundation.org
>>>>>>>         <mailto:Oisf-devel at openinfosecfoundation.org>
>>>>>>>
>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>     --
>>>>>>>     Regards,
>>>>>>>     Anoop Saldanha
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Anoop Saldanha
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------------------------
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Oisf-devel mailing list
>>>>>>> Oisf-devel at openinfosecfoundation.org
>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>>
>>>>>> --
>>>>>> ---------------------------------------------
>>>>>> Victor Julien
>>>>>> http://www.inliniac.net/
>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>>>> ---------------------------------------------
>>>>>>
>>>>>>
>>>>
>>>> -- 
>>>> ---------------------------------------------
>>>> Victor Julien
>>>> http://www.inliniac.net/
>>>> PGP: http://www.inliniac.net/victorjulien.asc
>>>> ---------------------------------------------
>>>>
>>>> _______________________________________________
>>>> Oisf-devel mailing list
>>>> Oisf-devel at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>
>>>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------




More information about the Oisf-devel mailing list