[Oisf-devel] Memory pb on Suricata git today
rmkml at free.fr
rmkml at free.fr
Fri Jul 30 21:13:47 UTC 2010
Thx you Éric for comment,
But my max-pending-packets is 50 by default.
Regards
Rmkml
Selon Eric Leblond <eleblond at edenwall.com>:
> Hi again,
>
> Le 30 juil. 2010 à 20:27, Eric Leblond <eleblond at edenwall.com> a écrit :
>
> > Hi,
> >
> > Le 30 juil. 2010 à 18:58, rmkml <rmkml at free.fr> a écrit :
> >
> >> Hi,
> >> Congratulations for Suricata v1.0.1!
> >> but this new release not fix my memory usage pb please.
> >> v101 - mem usage: 621M
> >> Im not continue my testing on your open source product because my linux
> kernel kill suricata process...
> >
> > I ve myself experimented a heavy memory usage of suricata. It is linked
> with the max-pending-packets.
> Missing words here:
> This is a suricata.yaml variable and can thus be decreased. But I ve seen a
> huge performance improvement when increase it.
>
> > During init suricata preallocate this amount of packets.
> > But each Packet structure is of size 80384 and this can cause a huge memory
> usage.
> >
> > BR
> >
> > Eric
> >
> >> Regards
> >> Rmkml
> >>
> >>
> >> On Mon, 26 Jul 2010, rmkml wrote:
> >>
> >>> It's ok, but with my commercial sig, suricata use 1.2G and killed by
> linux
> >>> kernel (on my personnal laptop).
> >>> Anyone test with vrt sigs please? (v2.8.5.3 or old)
> >>> Regards
> >>> Rmkml
> >>>
> >>>
> >>> On Mon, 26 Jul 2010, rmkml wrote:
> >>>
> >>>> Hi Victor,
> >>>> ok I have tested with theses suricata versions: (same conf, same pcap
> file
> >>>> is 27Mo)
> >>>> v100 - mem usage: 400M
> >>>> git13jul- mem usage: 400M
> >>>> git21jul- mem usage: 630M
> >>>> git25jul- mem usage: 649M
> >>>> All test with emerging all sigs daily
> >>>> (http://www.emergingthreats.net/rules/emerging-all.rules.zip)
> >>>> Anyone confirm increase 50% memory please?
> >>>> Regards
> >>>> Rmkml
> >>>>
> >>>>
> >>>> On Mon, 26 Jul 2010, Victor Julien wrote:
> >>>>
> >>>>> I think the increased mem usage is caused by fixing some accuracy
> >>>>> issues. As far as I can tell, it's not a bug of some kind.
> >>>>>
> >>>>> Cheers,
> >>>>> Victor
> >>>>>
> >>>>> rmkml wrote:
> >>>>>> Thx Anoop and Victor,
> >>>>>> ok crash/segfault fixed,
> >>>>>> but mem usage increase always exist on git
> >>>>>> c25921edf01c9f2d2e3c639037528ef5440c566e.
> >>>>>> Regards
> >>>>>> Rmkml
> >>>>>>
> >>>>>>
> >>>>>> On Sun, 25 Jul 2010, Victor Julien wrote:
> >>>>>>
> >>>>>>> Should be fixed in current master. Thanks guys!
> >>>>>>>
> >>>>>>> Anoop Saldanha wrote:
> >>>>>>>> Attached a new patch. Please don't apply the older one. Fixed a
> small
> >>>>>>>> typo in the unittest. It should pass now.
> >>>>>>>>
> >>>>>>>> On Sun, Jul 25, 2010 at 10:48 AM, Anoop Saldanha
> <poonaatsoc at gmail.com
> >>>>>>>> <mailto:poonaatsoc at gmail.com>> wrote:
> >>>>>>>>
> >>>>>>>> Hi rmkml. Can you please check it with this attached patch.
> >>>>>>>> Should
> >>>>>>>> fix it. Added an unittest to the patch as well.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Sun, Jul 25, 2010 at 1:21 AM, <rmkml at free.fr
> >>>>>>>> <mailto:rmkml at free.fr>> wrote:
> >>>>>>>>
> >>>>>>>> Ok Im found my "crash" sig:
> >>>>>>>> alert udp any any -> any any (msg:"crash"; byte_test:4,>,2,0;
> >>>>>>>> byte_jump:1,0,relative; sid:11; )
> >>>>>>>> Regards
> >>>>>>>> Rmkml
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Selon rmkml <rmkml at free.fr <mailto:rmkml at free.fr>>:
> >>>>>>>>
> >>>>>>>>> thx for reply Victor,
> >>>>>>>>> no problemo:
> >>>>>>>>>
> >>>>>>>>> ...
> >>>>>>>>> [20560] 24/7/2010 -- 16:23:13 - (detect.c:302) <Error>
> >>>>>>>> (DetectLoadSigFile) --
> >>>>>>>>> [ERRCODE: SC_ERR_INVALID_SIGNATURE(39)] - Error parsing
> >>>>>>>> signature "alert tcp
> >>>>>>>>> $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"WEB-PHP
> >>>>>>>> shoutbox.php
> >>>>>>>>> access"; flow:to_server,established;
> >>>>>>>> uricontent:"/shoutbox.php";
> >>>>>>>>> reference:nessus,11668; classtype:web-application-activity;
> >>>>>>>> sid:2142;
> >>>>>>>>> rev:1;)" from file /home/test/snort/rules/web-php.rules at
> >>>>>>>> line 94
> >>>>>>>>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594) <Error>
> >>>>>>>>> (DetectBytejumpSetup) -- [ERRCODE:
> >>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
> >>>>>>>>> preceding content or uricontent or pcre option
> >>>>>>>>> *** glibc detected ***
> >>>>>>>>> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul201
> >>>>>>>>> 0/src/.libs/suricata: corrupted double-linked list:
> >>>>>>>> 0x0a51dea8 ***
> >>>>>>>>> ======= Backtrace: =========
> >>>>>>>>> /lib/libc.so.6[0xa9d06d]
> >>>>>>>>> ...
> >>>>>>>>>
> >>>>>>>>> Regards
> >>>>>>>>> Rmkml
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Sat, 24 Jul 2010, Victor Julien wrote:
> >>>>>>>>>
> >>>>>>>>>> Can you share the signature this is happening with?
> >>>>>>>> Privately if you
> >>>>>>>>> prefer.
> >>>>>>>>>>
> >>>>>>>>>> Cheers,
> >>>>>>>>>> Victor
> >>>>>>>>>>
> >>>>>>>>>> rmkml wrote:
> >>>>>>>>>>> Hi Victor,
> >>>>>>>>>>> Thx for your work and your time on this project!
> >>>>>>>>>>>
> >>>>>>>>>>> I have "downloaded" (git clone) last Suricata version,
> >>>>>>>>>>> but I have a glibc error (git
> >>>>>>>> ead29dc6918f4524f1fae7e892d3f86dac117c0a):
> >>>>>>>>>>> ...
> >>>>>>>>>>> [20560] 24/7/2010 -- 16:23:14 - (detect-bytejump.c:594)
> >>>>>>>> <Error>
> >>>>>>>>>>> (DetectBytejumpSetup) -- [ERRCODE:
> >>>>>>>> SC_ERR_INVALID_SIGNATURE(39)] - No
> >>>>>>>>>>> preceding content or uricontent or pcre option
> >>>>>>>>>>> *** glibc detected ***
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata:
> >>>>>>>>
> >>>>>>>>>>> corrupted double-linked list: 0x0a51dea8 ***
> >>>>>>>>>>> ======= Backtrace: =========
> >>>>>>>>>>> /lib/libc.so.6[0xa9d06d]
> >>>>>>>>>>> /lib/libc.so.6[0xa9eb2b]
> >>>>>>>>>>> /lib/libc.so.6(cfree+0x90)[0xaa2430]
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807b0dd]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c04a]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x807c1fb]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x806586e]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x8065d4b]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804bc70]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> /lib/libc.so.6(__libc_start_main+0xe0)[0xa4cf70]
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata[0x804aa01]
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> ======= Memory map: ========
> >>>>>>>>>>> 0072c000-0073e000 r-xp 00000000 08:02 3700508
> >>>>>>>> /lib/libz.so.1.2.3
> >>>>>>>>>>> 0073e000-0073f000 rw-p 00011000 08:02 3700508
> >>>>>>>> /lib/libz.so.1.2.3
> >>>>>>>>>>> 00a18000-00a33000 r-xp 00000000 08:02 11817698
> >>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
> >>>>>>>>>>> 00a33000-00a34000 r--p 0001a000 08:02 11817698
> >>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
> >>>>>>>>>>> 00a34000-00a35000 rw-p 0001b000 08:02 11817698
> >>>>>>>> /lib/ld-2.6.so <http://ld-2.6.so>
> >>>>>>>>>>> 00a37000-00b85000 r-xp 00000000 08:02 11817699
> >>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
> >>>>>>>>>>> 00b85000-00b87000 r--p 0014e000 08:02 11817699
> >>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
> >>>>>>>>>>> 00b87000-00b88000 rw-p 00150000 08:02 11817699
> >>>>>>>> /lib/libc-2.6.so <http://libc-2.6.so>
> >>>>>>>>>>> 00b88000-00b8b000 rw-p 00000000 00:00 0
> >>>>>>>>>>> 00bbf000-00bd3000 r-xp 00000000 08:02 5434178
> >>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
> >>>>>>>>>>> 00bd3000-00bd4000 r--p 00013000 08:02 5434178
> >>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
> >>>>>>>>>>> 00bd4000-00bd5000 rw-p 00014000 08:02 5434178
> >>>>>>>> /lib/libpthread-2.6.so <http://libpthread-2.6.so>
> >>>>>>>>>>> 00bd5000-00bd7000 rw-p 00000000 00:00 0
> >>>>>>>>>>> 00bee000-00c17000 r-xp 00000000 08:02 2078837
> >>>>>>>> /usr/lib/libpcap.so.0.9.7
> >>>>>>>>>>> 00c17000-00c19000 rw-p 00028000 08:02 2078837
> >>>>>>>> /usr/lib/libpcap.so.0.9.7
> >>>>>>>>>>> 00c58000-00c7f000 r-xp 00000000 08:02 5434342
> >>>>>>>> /lib/libpcre.so.0.0.1
> >>>>>>>>>>> 00c7f000-00c80000 rw-p 00026000 08:02 5434342
> >>>>>>>> /lib/libpcre.so.0.0.1
> >>>>>>>>>>> 05db4000-05dbf000 r-xp 00000000 08:02 5434249
> >>>>>>>>>>> /lib/libgcc_s-4.1.2-20070925.so.1
> >>>>>>>>>>> 05dbf000-05dc0000 rw-p 0000a000 08:02 5434249
> >>>>>>>>>>> /lib/libgcc_s-4.1.2-20070925.so.1
> >>>>>>>>>>> 08048000-08100000 r-xp 00000000 08:02 1244073
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 08100000-08101000 rw-p 000b8000 08:02 1244073
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> /home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/src/.libs/suricata
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> 08101000-0a53d000 rw-p 00000000 00:00 0 [heap]
> >>>>>>>>>>> b7400000-b7421000 rw-p 00000000 00:00 0
> >>>>>>>>>>> b7421000-b7500000 ---p 00000000 00:00 0
> >>>>>>>>>>> b7594000-b771c000 rw-p 00000000 00:00 0
> >>>>>>>>>>> b771c000-b7737000 r-xp 00000000 08:02 11261710
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
> >>>>>>>>>>> b7737000-b7738000 rw-p 0001a000 08:02 11261710
> >>>>>>>>>>>
> >>>>>>>>
> >>>>>>>> /home/test/oisf_suricata_ids/yaml-0.1.3/src/.libs/libyaml-0.so.2.0.1
> >>>>>>>>>>> b7748000-b7749000 rw-p 00000000 00:00 0
> >>>>>>>>>>> b7749000-b7758000 r-xp 00000000 08:02 654980
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> b7758000-b7759000 rw-p 0000e000 08:02 654980
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
>
/home/test/oisf_suricata_ids/suricata-1.0.1pregit24jul2010/libhtp/htp/.libs/libhtp-0.2.so.1.0.2
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> b7759000-b775a000 rw-p 00000000 00:00 0
> >>>>>>>>>>> b775a000-b775b000 r-xp 00000000 00:00 0 [vdso]
> >>>>>>>>>>> bf96c000-bf98d000 rw-p 00000000 00:00 0 [stack]
> >>>>>>>>>>> Abandon
> >>>>>>>>>>>
> >>>>>>>>>>> Regards
> >>>>>>>>>>> Rmkml
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Sat, 24 Jul 2010, Victor Julien wrote:
> >>>>>>>>>>>
> >>>>>>>>>>>> rmkml at free.fr <mailto:rmkml at free.fr> wrote:
> >>>>>>>>>>>>> I have new:
> >>>>>>>>>>>>> On git 21 jul, mem usage pb appear, but I have a small
> >>>>>>>> (revert)
> >>>>>>>>>>>>> change "resolv"
> >>>>>>>>>>>>> my pb, Move (back) this Line on if loop /* content */:
> >>>>>>>>>>>>> PatternMatchPreparePopulateMpm(de_ctx, sh);
> >>>>>>>>>>>>> #line 1081 in src/detect-engine-mpm.c
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks Rmkml. At this point I don't think there is
> >>>>>>>> anything wrong in the
> >>>>>>>>>>>> code there. The changes were done to fix some accuracy
> >>>>>>>> issues we were
> >>>>>>>>>>>> seeing.
> >>>>>>>>>>>>
> >>>>>>>>>>>> I did cleanup the code a bit in the latest git master. I
> >>>>>>>> don't expect
> >>>>>>>>>>>> anything to change, but maybe you can try anyway :)
> >>>>>>>>>>>>
> >>>>>>>>>>>> Cheers,
> >>>>>>>>>>>> Victor
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> --
> >>>>>>>>>>>> ---------------------------------------------
> >>>>>>>>>>>> Victor Julien
> >>>>>>>>>>>> http://www.inliniac.net/
> >>>>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>>>>>>>>>> ---------------------------------------------
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> --
> >>>>>>>>>> ---------------------------------------------
> >>>>>>>>>> Victor Julien
> >>>>>>>>>> http://www.inliniac.net/
> >>>>>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>>>>>>>> ---------------------------------------------
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Oisf-devel mailing list
> >>>>>>>> Oisf-devel at openinfosecfoundation.org
> >>>>>>>> <mailto:Oisf-devel at openinfosecfoundation.org>
> >>>>>>>>
> >>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Regards,
> >>>>>>>> Anoop Saldanha
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Regards,
> >>>>>>>> Anoop Saldanha
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> ------------------------------------------------------------------------
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Oisf-devel mailing list
> >>>>>>>> Oisf-devel at openinfosecfoundation.org
> >>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> ---------------------------------------------
> >>>>>>> Victor Julien
> >>>>>>> http://www.inliniac.net/
> >>>>>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>>>>> ---------------------------------------------
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> ---------------------------------------------
> >>>>> Victor Julien
> >>>>> http://www.inliniac.net/
> >>>>> PGP: http://www.inliniac.net/victorjulien.asc
> >>>>> ---------------------------------------------
> >>>>>
> >>>>> _______________________________________________
> >>>>> Oisf-devel mailing list
> >>>>> Oisf-devel at openinfosecfoundation.org
> >>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >>>>>
> >>>>
> >>>>
> >>>
> >> _______________________________________________
> >> Oisf-devel mailing list
> >> Oisf-devel at openinfosecfoundation.org
> >> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> > _______________________________________________
> > Oisf-devel mailing list
> > Oisf-devel at openinfosecfoundation.org
> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
More information about the Oisf-devel
mailing list