If you're up for it, about 15 lines of C code will give you a tiny program to test how much memory you can get for a single process - basically just malloc in a loop until you can't anymore. Sounds like your environment may actually be limited to 2GB of process size; normal for Linux is 3GB (all in the 32 bit world). Or you could lobby for a 64 bit copy<div>
of Centos; that'll eliminate the cap (for this purpose).</div><div><br></div><div>Cheers,</div><div><br></div><div>Dave<br><br><div class="gmail_quote">On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <span dir="ltr"><<a href="mailto:gene.albin@gmail.com">gene.albin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">I just created a ticket with the details. To answer the questions here, I'm running the 1.1b2 build from the tarball. Not using git. The machine is running the 32 bit version of CentOS5.6, but we just applied the kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is this what you are talking about, Dave? Lastly I included the suricata.yaml file as well as the output from free -m and my collectl memory statistics during the fatal run.<br>
<br>Thanks for helping out with this. I thought that bumping the ram up to 16GB would fix it, but it appears not. Maybe I'll start slicing off some rules and see where the threshold lies...<br><font color="#888888"><br>
Gene</font><div><div></div><div class="h5"><br><br><div class="gmail_quote">
On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <span dir="ltr"><<a href="mailto:dave.remien@gmail.com" target="_blank">dave.remien@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><br><div class="gmail_quote"><div>On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <span dir="ltr"><<a href="mailto:william.metcalf@gmail.com" target="_blank">william.metcalf@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Can you create a redmine ticket and attach a scrubbed version of your<br>
suricata.yaml? Along with output of free -m prior to starting suri?<br></blockquote><div><br></div></div><div>Are you running a 32 bit kernel with a 2GB/2GB memory split, by any chance??</div><div><br></div><div>Cheers,</div>
<div><br></div><div>Dave</div><div><div></div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<a href="https://redmine.openinfosecfoundation.org/projects/suricata" target="_blank">https://redmine.openinfosecfoundation.org/projects/suricata</a><br>
<br>
Regards,<br>
<br>
Will<br>
<div><div></div><div>On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>> wrote:<br>
> Ok, I'm probably doing something wrong here, but every time I try to load a<br>
> combined rule file with all of the VRT and ET rules enabled (~30K rules) it<br>
> fails following stage 3:<br>
><br>
> [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info> (SigLoadSignatures) --<br>
> 102 rule files processed. 30183 rules succesfully loaded, 164 rules failed<br>
> [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info><br>
> (SigAddressPrepareStage1) -- 30701 signatures processed. 1800 are IP-only<br>
> rules, 20152 are inspecting packet payload, 11088 inspect application layer,<br>
> 0 are decoder event only<br>
> [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info><br>
> (SigAddressPrepareStage1) -- building signature grouping structure, stage 1:<br>
> adding signatures to signature source addresses... complete<br>
> [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info><br>
> (SigAddressPrepareStage2) -- building signature grouping structure, stage 2:<br>
> building source address list... complete<br>
> [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info><br>
> (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic 1801173581, ctxs<br>
> 0, avg per ctx 0)<br>
> [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info><br>
> (SigAddressPrepareStage3) -- max sig id 30701, array size 3838<br>
> [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info><br>
> (SigAddressPrepareStage3) -- building signature grouping structure, stage 3:<br>
> building destination address lists... complete<br>
> [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error><br>
> (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - SCMalloc<br>
> failed: Cannot allocate memory, while trying to allocate 558852 bytes<br>
><br>
> [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error><br>
> (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] - Out of<br>
> memory. The engine cannot be initialized. Exiting...<br>
><br>
> I have done this while watching the memory useage in top (set to refresh<br>
> every .2 seconds). Initially when this happened I only had 4GB allocated to<br>
> the VM. Useage never gets beyond 2GB so that left almost 2GB available. I<br>
> decided to bump the VM up to 8GB but the problem didn't go away. It still<br>
> exits when the memory useage gets to around 2GB.<br>
><br>
> Everything works fine when I load a reduced ruleset, i.e. just VRT or just<br>
> ET, but for my tests I want to load both. Before I go back to the VM<br>
> administrator and ask for 16 GB (and wait several days for the allocation) I<br>
> was wondering if there might be a config setting that is limiting the size<br>
> of memory allocated to the rules.<br>
><br>
> Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.<br>
><br>
> Any suggestions are welcome.<br>
><br>
> Gene<br>
><br>
> --<br>
> Gene Albin<br>
> <a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Oisf-users mailing list<br>
> <a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
> <a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
><br>
><br>
_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
</blockquote></div></div></div><font color="#888888"><br><br clear="all"><br>-- <br>"Of course, someone who knows more about this will correct me if I'm<br>wrong, and someone who knows less will correct me if I'm right." <br>
David Palmer (<a href="mailto:palmer@tybalt.caltech.edu" target="_blank">palmer@tybalt.caltech.edu</a>)<br>
<br>
</font></blockquote></div><br><br clear="all"><br></div></div>-- <br><div><div></div><div class="h5">Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>"Of course, someone who knows more about this will correct me if I'm<br>wrong, and someone who knows less will correct me if I'm right." <br>
David Palmer (<a href="mailto:palmer@tybalt.caltech.edu">palmer@tybalt.caltech.edu</a>)<br><br>
</div>