Peter,<div><br></div><div>Gene has a 1G/3G kernel and can indeed get 3G; I sent him a quickie prog that shows that. After that, seems like valgrind is in order to try to figure out where the extra memory is being allocated, if you can't see why from the suri.yaml files.... possibly an area Victor could shed light on. Or, as you say, move to a 64 bit kernel. At least there Gene can get 4G - 8-).</div>
<div><br></div><div>Cheers,</div><div><br></div><div>Dave<br><br><div class="gmail_quote">On Sun, Jul 31, 2011 at 5:09 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><div><div></div><div class="h5"><br><br><div class="gmail_quote">On Sat, Jul 30, 2011 at 5:49 PM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div><div></div><div><br><br><div class="gmail_quote">On Fri, Jul 29, 2011 at 5:34 PM, Gene Albin <span dir="ltr"><<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
Peter,<br>
Sudo suricata ends with the same problem. I'm wondering if there
may be a difference between the optimization of our suricata.yaml
files. Would you mind exchanging files so that we can compare?<br><font color="#888888">
<br>
Gene</font><div><div></div><div><br><br><div class="gmail_quote">On Fri, Jul 29, 2011 at 1:05 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<div><div></div><div><br><br><div class="gmail_quote">On Thu, Jul 28, 2011 at 8:59 PM, Gene Albin <span dir="ltr"><<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
Gents,<br> Thanks all for the suggestions. Specific responses below<br> <br>Rmkml - I made your recommended change to the suricata.yaml file (- profile: low) and there was no change. Watching top the spike appeard to peak at 3.7GB then Suricata exited.<br>
<br>Dave - I have to admit that I'm not versed in C so I can't build the program you're describing. Do you know where I could find a pre-built one? I agree that upgrading to a 64-bit version would be the best choice, however I'm up against a very short timeline here and won't have time to upgrade the OS the reinstall Suricata and the rest of the tools that I need. <br>
<br>Peter - Here are the results from both ulimit -aH and ulimit -a:<br><br><span style="font-family:courier new,monospace">[gene@suri2 ~]$ ulimit -aH</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">core file size (blocks, -c) unlimited</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">data seg size (kbytes, -d) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">scheduling priority (-e) 0</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">file size (blocks, -f) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">pending signals (-i) 278528</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">max locked memory (kbytes, -l) 32</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">max memory size (kbytes, -m) unlimited</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">open files (-n) 1024</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">pipe size (512 bytes, -p) 8</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">POSIX message queues (bytes, -q) 819200</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">real-time priority (-r) 0</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">stack size (kbytes, -s) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">cpu time (seconds, -t) unlimited</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">max user processes (-u) 278528</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">virtual memory (kbytes, -v) unlimited</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">file locks (-x) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace"><br>[gene@suri2 ~]$ ulimit -a</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">core file size (blocks, -c) 0</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">data seg size (kbytes, -d) unlimited</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">scheduling priority (-e) 0</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">file size (blocks, -f) unlimited</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">pending signals (-i) 278528</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">max locked memory (kbytes, -l) 32</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">max memory size (kbytes, -m) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">open files (-n) 1024</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">pipe size (512 bytes, -p) 8</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">POSIX message queues (bytes, -q) 819200</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">real-time priority (-r) 0</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">stack size (kbytes, -s) 10240</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">cpu time (seconds, -t) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">max user processes (-u) 278528</span><br style="font-family:courier new,monospace">
<span style="font-family:courier new,monospace">virtual memory (kbytes, -v) unlimited</span><br style="font-family:courier new,monospace"><span style="font-family:courier new,monospace">file locks (-x) unlimited</span><br>
<br>I'm not exactly sure what I'm looking at here, but it looks like my max memory size is unlimited. Am I reading this correctly?<br>I've also looked over your screen capture from your Ubuntu 32-bit VM and can't figure out why I'm crashing. What version of Suricata are you running on that 32-bit Ubuntu VM? 1.1b2?<br>
<font color="#888888">
<br>Gene</font><div><div></div><div><br><br><div class="gmail_quote">On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <span dir="ltr"><<a href="mailto:petermanev@gmail.com" target="_blank">petermanev@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<u></u>
<div text="#000000" bgcolor="#ffffff"><div><div></div><div>
On 07/28/2011 02:54 PM, Dave Remien wrote:
<blockquote type="cite">If you're up for it, about 15 lines of C code will
give you a tiny program to test how much memory you can get for a
single process - basically just malloc in a loop until you can't
anymore. Sounds like your environment may actually be limited to
2GB of process size; normal for Linux is 3GB (all in the 32 bit
world). Or you could lobby for a 64 bit copy
<div>
of Centos; that'll eliminate the cap (for this purpose).</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Dave<br>
<br>
<div class="gmail_quote">On Thu, Jul 28, 2011 at 1:10 AM, Gene
Albin <span dir="ltr"><<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">I just created a ticket with the
details. To answer the questions here, I'm running the
1.1b2 build from the tarball. Not using git. The machine
is running the 32 bit version of CentOS5.6, but we just
applied the kernel-PAE packages today to allow it to utilize
more than 4GB of ram. Is this what you are talking about,
Dave? Lastly I included the suricata.yaml file as well as
the output from free -m and my collectl memory statistics
during the fatal run.<br>
<br>
Thanks for helping out with this. I thought that bumping
the ram up to 16GB would fix it, but it appears not. Maybe
I'll start slicing off some rules and see where the
threshold lies...<br>
<font color="#888888"><br>
Gene</font>
<div>
<div><br>
<br>
<div class="gmail_quote">
On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <span dir="ltr"><<a href="mailto:dave.remien@gmail.com" target="_blank">dave.remien@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<br>
<br>
<div class="gmail_quote">
<div>On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf
<span dir="ltr"><<a href="mailto:william.metcalf@gmail.com" target="_blank">william.metcalf@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
Can you create a redmine ticket and attach a
scrubbed version of your<br>
suricata.yaml? Along with output of free -m
prior to starting suri?<br>
</blockquote>
<div><br>
</div>
</div>
<div>Are you running a 32 bit kernel with a
2GB/2GB memory split, by any chance??</div>
<div><br>
</div>
<div>Cheers,</div>
<div><br>
</div>
<div>Dave</div>
<div>
<div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex">
<br>
<a href="https://redmine.openinfosecfoundation.org/projects/suricata" target="_blank">https://redmine.openinfosecfoundation.org/projects/suricata</a><br>
<br>
Regards,<br>
<br>
Will<br>
<div>
<div>On Wed, Jul 27, 2011 at 4:35 PM, Gene
Albin <<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a>>
wrote:<br>
> Ok, I'm probably doing something
wrong here, but every time I try to load
a<br>
> combined rule file with all of the
VRT and ET rules enabled (~30K rules) it<br>
> fails following stage 3:<br>
><br>
> [7069] 27/7/2011 -- 14:14:09 -
(detect.c:631) <Info>
(SigLoadSignatures) --<br>
> 102 rule files processed. 30183
rules succesfully loaded, 164 rules
failed<br>
> [7069] 27/7/2011 -- 14:14:47 -
(detect.c:2161) <Info><br>
> (SigAddressPrepareStage1) -- 30701
signatures processed. 1800 are IP-only<br>
> rules, 20152 are inspecting packet
payload, 11088 inspect application
layer,<br>
> 0 are decoder event only<br>
> [7069] 27/7/2011 -- 14:14:47 -
(detect.c:2164) <Info><br>
> (SigAddressPrepareStage1) --
building signature grouping structure,
stage 1:<br>
> adding signatures to signature
source addresses... complete<br>
> [7069] 27/7/2011 -- 14:14:48 -
(detect.c:2806) <Info><br>
> (SigAddressPrepareStage2) --
building signature grouping structure,
stage 2:<br>
> building source address list...
complete<br>
> [7069] 27/7/2011 -- 14:16:40 -
(detect.c:3363) <Info><br>
> (SigAddressPrepareStage3) -- MPM
memory 1801173581 (dynamic 1801173581,
ctxs<br>
> 0, avg per ctx 0)<br>
> [7069] 27/7/2011 -- 14:16:40 -
(detect.c:3365) <Info><br>
> (SigAddressPrepareStage3) -- max
sig id 30701, array size 3838<br>
> [7069] 27/7/2011 -- 14:16:40 -
(detect.c:3376) <Info><br>
> (SigAddressPrepareStage3) --
building signature grouping structure,
stage 3:<br>
> building destination address
lists... complete<br>
> [7069] 27/7/2011 -- 14:16:43 -
(detect-engine-siggroup.c:1583)
<Error><br>
> (SigGroupHeadBuildHeadArray) --
[ERRCODE: SC_ERR_MEM_ALLOC(1)] -
SCMalloc<br>
> failed: Cannot allocate memory,
while trying to allocate 558852 bytes<br>
><br>
> [7069] 27/7/2011 -- 14:16:43 -
(detect-engine-siggroup.c:1583)
<Error><br>
> (SigGroupHeadBuildHeadArray) --
[ERRCODE: SC_ERR_FATAL(169)] - Out of<br>
> memory. The engine cannot be
initialized. Exiting...<br>
><br>
> I have done this while watching the
memory useage in top (set to refresh<br>
> every .2 seconds). Initially when
this happened I only had 4GB allocated
to<br>
> the VM. Useage never gets beyond
2GB so that left almost 2GB available.
I<br>
> decided to bump the VM up to 8GB
but the problem didn't go away. It
still<br>
> exits when the memory useage gets
to around 2GB.<br>
><br>
> Everything works fine when I load a
reduced ruleset, i.e. just VRT or just<br>
> ET, but for my tests I want to load
both. Before I go back to the VM<br>
> administrator and ask for 16 GB
(and wait several days for the
allocation) I<br>
> was wondering if there might be a
config setting that is limiting the size<br>
> of memory allocated to the rules.<br>
><br>
> Running 1.1b2 on CentOS 5.6 - 4core
VMWare ESXi.<br>
><br>
> Any suggestions are welcome.<br>
><br>
> Gene<br>
><br>
> --<br>
> Gene Albin<br>
> <a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
><br>
><br>
</div>
</div>
>
_______________________________________________<br>
> Oisf-users mailing list<br>
> <a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
> <a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
><br>
><br>
_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
</blockquote>
</div>
</div>
</div>
<font color="#888888"><br>
<br clear="all">
<br>
-- <br>
"Of course, someone who knows more about this will
correct me if I'm<br>
wrong, and someone who knows less will correct me
if I'm right." <br>
David Palmer (<a href="mailto:palmer@tybalt.caltech.edu" target="_blank">palmer@tybalt.caltech.edu</a>)<br>
<br>
</font></blockquote>
</div>
<br>
<br clear="all">
<br>
</div>
</div>
-- <br>
<div>
<div>Gene Albin<br>
<a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br>
<br>
</div>
</div>
</blockquote>
</div>
<br>
<br clear="all">
<br>
-- <br>
"Of course, someone who knows more about this will correct me if
I'm<br>
wrong, and someone who knows less will correct me if I'm right."
<br>
David Palmer (<a href="mailto:palmer@tybalt.caltech.edu" target="_blank">palmer@tybalt.caltech.edu</a>)<br>
<br>
</div>
<pre><fieldset></fieldset>
_______________________________________________
Oisf-users mailing list
<a href="mailto:Oisf-users@openinfosecfoundation.org" target="_blank">Oisf-users@openinfosecfoundation.org</a>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a>
</pre>
</blockquote></div></div>
In that respect.... What is your output of <br>
ulimit -aH <br>
and<br>
ulimit -a<br>
for the user that you run Suricata with?<br>
<br>
<pre cols="72">--
Regards,
Peter Manev</pre>
</div>
</blockquote></div><br><br clear="all"><br></div></div>-- <br><div><div></div><div>Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</div></div></blockquote></div><br><br><br></div></div>Hi Gene, <br>I am using Suricata version 1.1beta2 (rev df3ca32).<br>What happens if you try to run Suricata as root?<br><br>Thanks<br><br clear="all"><br>-- <br><font color="#888888">Peter Manev<br>
</font></blockquote></div><br><br clear="all"><br></div></div>-- <br><div><div></div><div>Gene Albin<br><a href="mailto:gene.albin@gmail.com" target="_blank">gene.albin@gmail.com</a><br><br>
</div></div></blockquote></div><br><br clear="all"><br></div></div>Sure,<br>I did send mine to you.<br>I tried with yours and Suricata is running aswell (I only modified the rules directory)<br><br>Thanks<br><br>-- <br><font color="#888888">Peter Manev<br>
</font></blockquote></div><br></div></div>Hi Gene<br clear="all"><br>I think Dave is right - you might be running 2Gb/2Gb memory split on a 32bit.<br>I am running 1G / 3G memory split which gives me the edge I believe.<br>
I am inclined to believe(please correct me if I am wrong ) that you have a 2Gb per process limit because of that (since you said yourself that Suricata always gets killed when it reaches 2Gb of use at start up ) the PAE is not going to help you in this case - if you would like to check that for sure you could try 64bit system.<br>
<br>Thanks<br>-- <br><font color="#888888">Peter Manev<br>
</font></blockquote></div><br><br clear="all"><br>-- <br>"Of course, someone who knows more about this will correct me if I'm<br>wrong, and someone who knows less will correct me if I'm right." <br>David Palmer (<a href="mailto:palmer@tybalt.caltech.edu">palmer@tybalt.caltech.edu</a>)<br>
<br>
</div>