[Oisf-users] Suricata runs out of memory on startup

[Gene Albin]

I just created a ticket with the details.  To answer the questions here, I'm
running the 1.1b2 build from the tarball.  Not using git.  The machine is
running the 32 bit version of CentOS5.6, but we just applied the kernel-PAE
packages today to allow it to utilize more than 4GB of ram.  Is this what
you are talking about, Dave?  Lastly I included the suricata.yaml file as
well as the output from free -m and my collectl memory statistics during the
fatal run.

Thanks for helping out with this.  I thought that bumping the ram up to 16GB
would fix it, but it appears not.  Maybe I'll start slicing off some rules
and see where the threshold lies...

Gene

On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <dave.remien at gmail.com> wrote:

>
>
> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <william.metcalf at gmail.com>wrote:
>
>> Can you create a redmine ticket and attach a scrubbed version of your
>> suricata.yaml?  Along with output of free -m prior to starting suri?
>>
>
> Are you running a 32 bit kernel with a 2GB/2GB memory split, by any
> chance??
>
> Cheers,
>
> Dave
>
>
>>
>> https://redmine.openinfosecfoundation.org/projects/suricata
>>
>> Regards,
>>
>> Will
>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <gene.albin at gmail.com> wrote:
>> > Ok,  I'm probably doing something wrong here, but every time I try to
>> load a
>> > combined rule file with all of the VRT and ET rules enabled (~30K rules)
>> it
>> > fails following stage 3:
>> >
>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info> (SigLoadSignatures)
>> --
>> > 102 rule files processed. 30183 rules succesfully loaded, 164 rules
>> failed
>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800 are
>> IP-only
>> > rules, 20152 are inspecting packet payload, 11088 inspect application
>> layer,
>> > 0 are decoder event only
>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>> > (SigAddressPrepareStage1) -- building signature grouping structure,
>> stage 1:
>> > adding signatures to signature source addresses... complete
>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>> > (SigAddressPrepareStage2) -- building signature grouping structure,
>> stage 2:
>> > building source address list... complete
>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic 1801173581,
>> ctxs
>> > 0, avg per ctx 0)
>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>> > (SigAddressPrepareStage3) -- building signature grouping structure,
>> stage 3:
>> > building destination address lists... complete
>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error>
>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] -
>> SCMalloc
>> > failed: Cannot allocate memory, while trying to allocate 558852 bytes
>> >
>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error>
>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] - Out of
>> > memory. The engine cannot be initialized. Exiting...
>> >
>> > I have done this while watching the memory useage in top (set to refresh
>> > every .2 seconds).  Initially when this happened I only had 4GB
>> allocated to
>> > the VM.  Useage never gets beyond 2GB so that left almost 2GB
>> available.  I
>> > decided to bump the VM up to 8GB but the problem didn't go away.  It
>> still
>> > exits when the memory useage gets to around 2GB.
>> >
>> > Everything works fine when I load a reduced ruleset, i.e. just VRT or
>> just
>> > ET, but for my tests I want to load both.  Before I go back to the VM
>> > administrator and ask for 16 GB (and wait several days for the
>> allocation) I
>> > was wondering if there might be a config setting that is limiting the
>> size
>> > of memory allocated to the rules.
>> >
>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>> >
>> > Any suggestions are welcome.
>> >
>> > Gene
>> >
>> > --
>> > Gene Albin
>> > gene.albin at gmail.com
>> >
>> >
>> > _______________________________________________
>> > Oisf-users mailing list
>> > Oisf-users at openinfosecfoundation.org
>> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>> >
>> >
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>
>
>
> --
> "Of course, someone who knows more about this will correct me if I'm
> wrong, and someone who knows less will correct me if I'm right."
> David Palmer (palmer at tybalt.caltech.edu)
>
>


-- 
Gene Albin
gene.albin at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110728/728f9027/attachment-0002.html>