<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body text="#000000" bgcolor="#ffffff">
On 07/28/2011 02:54 PM, Dave Remien wrote:
<blockquote
cite="mid:CAD8uqfAbzvKUxfU46DwTAnA2RSK41inczcwg0EJG-L9yCA-Z-g@mail.gmail.com"
type="cite">If you're up for it, about 15 lines of C code will
give you a tiny program to test how much memory you can get for a
single process - basically just malloc in a loop until you can't
anymore. Sounds like your environment may actually be limited to
2GB of process size; normal for Linux is 3GB (all in the 32 bit
world). Or you could lobby for a 64 bit copy
<div>
of Centos; that'll eliminate the cap (for this purpose).</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Dave<br>
<br>
<div class="gmail_quote">On Thu, Jul 28, 2011 at 1:10 AM, Gene
Albin <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:gene.albin@gmail.com">gene.albin@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt
0.8ex; border-left: 1px solid rgb(204, 204, 204);
padding-left: 1ex;">I just created a ticket with the
details. To answer the questions here, I'm running the
1.1b2 build from the tarball. Not using git. The machine
is running the 32 bit version of CentOS5.6, but we just
applied the kernel-PAE packages today to allow it to utilize
more than 4GB of ram. Is this what you are talking about,
Dave? Lastly I included the suricata.yaml file as well as
the output from free -m and my collectl memory statistics
during the fatal run.<br>
<br>
Thanks for helping out with this. I thought that bumping
the ram up to 16GB would fix it, but it appears not. Maybe
I'll start slicing off some rules and see where the
threshold lies...<br>
<font color="#888888"><br>
Gene</font>
<div>
<div class="h5"><br>
<br>
<div class="gmail_quote">
On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:dave.remien@gmail.com"
target="_blank">dave.remien@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt
0pt 0.8ex; border-left: 1px solid rgb(204, 204,
204); padding-left: 1ex;">
<br>
<br>
<div class="gmail_quote">
<div>On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf
<span dir="ltr"><<a moz-do-not-send="true"
href="mailto:william.metcalf@gmail.com"
target="_blank">william.metcalf@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:
0pt 0pt 0pt 0.8ex; border-left: 1px solid
rgb(204, 204, 204); padding-left: 1ex;">
Can you create a redmine ticket and attach a
scrubbed version of your<br>
suricata.yaml? Along with output of free -m
prior to starting suri?<br>
</blockquote>
<div><br>
</div>
</div>
<div>Are you running a 32 bit kernel with a
2GB/2GB memory split, by any chance??</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Dave</div>
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:
0pt 0pt 0pt 0.8ex; border-left: 1px solid
rgb(204, 204, 204); padding-left: 1ex;">
<br>
<a moz-do-not-send="true"
href="https://redmine.openinfosecfoundation.org/projects/suricata"
target="_blank">https://redmine.openinfosecfoundation.org/projects/suricata</a><br>
<br>
Regards,<br>
<br>
Will<br>
<div>
<div>On Wed, Jul 27, 2011 at 4:35 PM, Gene
Albin <<a moz-do-not-send="true"
href="mailto:gene.albin@gmail.com"
target="_blank">gene.albin@gmail.com</a>>
wrote:<br>
> Ok, I'm probably doing something
wrong here, but every time I try to load
a<br>
> combined rule file with all of the
VRT and ET rules enabled (~30K rules) it<br>
> fails following stage 3:<br>
><br>
> [7069] 27/7/2011 -- 14:14:09 -
(detect.c:631) <Info>
(SigLoadSignatures) --<br>
> 102 rule files processed. 30183
rules succesfully loaded, 164 rules
failed<br>
> [7069] 27/7/2011 -- 14:14:47 -
(detect.c:2161) <Info><br>
> (SigAddressPrepareStage1) -- 30701
signatures processed. 1800 are IP-only<br>
> rules, 20152 are inspecting packet
payload, 11088 inspect application
layer,<br>
> 0 are decoder event only<br>
> [7069] 27/7/2011 -- 14:14:47 -
(detect.c:2164) <Info><br>
> (SigAddressPrepareStage1) --
building signature grouping structure,
stage 1:<br>
> adding signatures to signature
source addresses... complete<br>
> [7069] 27/7/2011 -- 14:14:48 -
(detect.c:2806) <Info><br>
> (SigAddressPrepareStage2) --
building signature grouping structure,
stage 2:<br>
> building source address list...
complete<br>
> [7069] 27/7/2011 -- 14:16:40 -
(detect.c:3363) <Info><br>
> (SigAddressPrepareStage3) -- MPM
memory 1801173581 (dynamic 1801173581,
ctxs<br>
> 0, avg per ctx 0)<br>
> [7069] 27/7/2011 -- 14:16:40 -
(detect.c:3365) <Info><br>
> (SigAddressPrepareStage3) -- max
sig id 30701, array size 3838<br>
> [7069] 27/7/2011 -- 14:16:40 -
(detect.c:3376) <Info><br>
> (SigAddressPrepareStage3) --
building signature grouping structure,
stage 3:<br>
> building destination address
lists... complete<br>
> [7069] 27/7/2011 -- 14:16:43 -
(detect-engine-siggroup.c:1583)
<Error><br>
> (SigGroupHeadBuildHeadArray) --
[ERRCODE: SC_ERR_MEM_ALLOC(1)] -
SCMalloc<br>
> failed: Cannot allocate memory,
while trying to allocate 558852 bytes<br>
><br>
> [7069] 27/7/2011 -- 14:16:43 -
(detect-engine-siggroup.c:1583)
<Error><br>
> (SigGroupHeadBuildHeadArray) --
[ERRCODE: SC_ERR_FATAL(169)] - Out of<br>
> memory. The engine cannot be
initialized. Exiting...<br>
><br>
> I have done this while watching the
memory useage in top (set to refresh<br>
> every .2 seconds). Initially when
this happened I only had 4GB allocated
to<br>
> the VM. Useage never gets beyond
2GB so that left almost 2GB available.
I<br>
> decided to bump the VM up to 8GB
but the problem didn't go away. It
still<br>
> exits when the memory useage gets
to around 2GB.<br>
><br>
> Everything works fine when I load a
reduced ruleset, i.e. just VRT or just<br>
> ET, but for my tests I want to load
both. Before I go back to the VM<br>
> administrator and ask for 16 GB
(and wait several days for the
allocation) I<br>
> was wondering if there might be a
config setting that is limiting the size<br>
> of memory allocated to the rules.<br>
><br>
> Running 1.1b2 on CentOS 5.6 - 4core
VMWare ESXi.<br>
><br>
> Any suggestions are welcome.<br>
><br>
> Gene<br>
><br>
> --<br>
> Gene Albin<br>
> <a moz-do-not-send="true"
href="mailto:gene.albin@gmail.com"
target="_blank">gene.albin@gmail.com</a><br>
><br>
><br>
</div>
</div>
>
_______________________________________________<br>
> Oisf-users mailing list<br>
> <a moz-do-not-send="true"
href="mailto:Oisf-users@openinfosecfoundation.org"
target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
> <a moz-do-not-send="true"
href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users"
target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
><br>
><br>
_______________________________________________<br>
Oisf-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Oisf-users@openinfosecfoundation.org"
target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users"
target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
</blockquote>
</div>
</div>
</div>
<font color="#888888"><br>
<br clear="all">
<br>
-- <br>
"Of course, someone who knows more about this will
correct me if I'm<br>
wrong, and someone who knows less will correct me
if I'm right." <br>
David Palmer (<a moz-do-not-send="true"
href="mailto:palmer@tybalt.caltech.edu"
target="_blank">palmer@tybalt.caltech.edu</a>)<br>
<br>
</font></blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
-- <br>
<div>
<div class="h5">Gene Albin<br>
<a moz-do-not-send="true"
href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
"Of course, someone who knows more about this will correct me if
I'm<br>
wrong, and someone who knows less will correct me if I'm right."
<br>
David Palmer (<a moz-do-not-send="true"
href="mailto:palmer@tybalt.caltech.edu">palmer@tybalt.caltech.edu</a>)<br>
<br>
</div>
<pre wrap="">
<fieldset class="mimeAttachmentHeader"></fieldset>
_______________________________________________
Oisf-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a>
</pre>
</blockquote>
In that respect.... What is your output of <br>
ulimit -aH <br>
and<br>
ulimit -a<br>
for the user that you run Suricata with?<br>
<br>
<pre class="moz-signature" cols="72">--
Regards,
Peter Manev</pre>
</body>
</html>