[Oisf-users] Suricata runs out of memory on startup

[Gene Albin]

Ok,  I'm probably doing something wrong here, but every time I try to load a
combined rule file with all of the VRT and ET rules enabled (~30K rules) it
fails following stage 3:

[7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info> (SigLoadSignatures) --
102 rule files processed. 30183 rules succesfully loaded, 164 rules failed
[7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
(SigAddressPrepareStage1) -- 30701 signatures processed. 1800 are IP-only
rules, 20152 are inspecting packet payload, 11088 inspect application layer,
0 are decoder event only
[7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
(SigAddressPrepareStage1) -- building signature grouping structure, stage 1:
adding signatures to signature source addresses... complete
[7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
(SigAddressPrepareStage2) -- building signature grouping structure, stage 2:
building source address list... complete
[7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
(SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic 1801173581, ctxs
0, avg per ctx 0)
[7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
(SigAddressPrepareStage3) -- max sig id 30701, array size 3838
[7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
(SigAddressPrepareStage3) -- building signature grouping structure, stage 3:
building destination address lists... complete
[7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error>
(SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - SCMalloc
failed: Cannot allocate memory, while trying to allocate 558852 bytes

[7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error>
(SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] - Out of
memory. The engine cannot be initialized. Exiting...

I have done this while watching the memory useage in top (set to refresh
every .2 seconds).  Initially when this happened I only had 4GB allocated to
the VM.  Useage never gets beyond 2GB so that left almost 2GB available.  I
decided to bump the VM up to 8GB but the problem didn't go away.  It still
exits when the memory useage gets to around 2GB.

Everything works fine when I load a reduced ruleset, i.e. just VRT or just
ET, but for my tests I want to load both.  Before I go back to the VM
administrator and ask for 16 GB (and wait several days for the allocation) I
was wondering if there might be a config setting that is limiting the size
of memory allocated to the rules.

Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.

Any suggestions are welcome.

Gene

-- 
Gene Albin
gene.albin at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110727/2225b894/attachment-0002.html>