[Oisf-users] Suricata runs out of memory on startup
Gene Albin
gene.albin at gmail.com
Thu Jul 28 18:59:47 UTC 2011
Gents,
Thanks all for the suggestions. Specific responses below
Rmkml - I made your recommended change to the suricata.yaml file (- profile:
low) and there was no change. Watching top the spike appeard to peak at
3.7GB then Suricata exited.
Dave - I have to admit that I'm not versed in C so I can't build the program
you're describing. Do you know where I could find a pre-built one? I agree
that upgrading to a 64-bit version would be the best choice, however I'm up
against a very short timeline here and won't have time to upgrade the OS the
reinstall Suricata and the rest of the tools that I need.
Peter - Here are the results from both ulimit -aH and ulimit -a:
[gene at suri2 ~]$ ulimit -aH
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 278528
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) unlimited
cpu time (seconds, -t) unlimited
max user processes (-u) 278528
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
[gene at suri2 ~]$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 278528
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 278528
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
I'm not exactly sure what I'm looking at here, but it looks like my max
memory size is unlimited. Am I reading this correctly?
I've also looked over your screen capture from your Ubuntu 32-bit VM and
can't figure out why I'm crashing. What version of Suricata are you running
on that 32-bit Ubuntu VM? 1.1b2?
Gene
On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <petermanev at gmail.com> wrote:
> **
> On 07/28/2011 02:54 PM, Dave Remien wrote:
>
> If you're up for it, about 15 lines of C code will give you a tiny program
> to test how much memory you can get for a single process - basically just
> malloc in a loop until you can't anymore. Sounds like your environment may
> actually be limited to 2GB of process size; normal for Linux is 3GB (all in
> the 32 bit world). Or you could lobby for a 64 bit copy
> of Centos; that'll eliminate the cap (for this purpose).
>
> Cheers,
>
> Dave
>
> On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <gene.albin at gmail.com> wrote:
>
>> I just created a ticket with the details. To answer the questions here,
>> I'm running the 1.1b2 build from the tarball. Not using git. The machine
>> is running the 32 bit version of CentOS5.6, but we just applied the
>> kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is
>> this what you are talking about, Dave? Lastly I included the suricata.yaml
>> file as well as the output from free -m and my collectl memory statistics
>> during the fatal run.
>>
>> Thanks for helping out with this. I thought that bumping the ram up to
>> 16GB would fix it, but it appears not. Maybe I'll start slicing off some
>> rules and see where the threshold lies...
>>
>> Gene
>>
>>
>> On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <dave.remien at gmail.com>wrote:
>>
>>>
>>>
>>> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <
>>> william.metcalf at gmail.com> wrote:
>>>
>>>> Can you create a redmine ticket and attach a scrubbed version of your
>>>> suricata.yaml? Along with output of free -m prior to starting suri?
>>>>
>>>
>>> Are you running a 32 bit kernel with a 2GB/2GB memory split, by any
>>> chance??
>>>
>>> Cheers,
>>>
>>> Dave
>>>
>>>
>>>>
>>>> https://redmine.openinfosecfoundation.org/projects/suricata
>>>>
>>>> Regards,
>>>>
>>>> Will
>>>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <gene.albin at gmail.com>
>>>> wrote:
>>>> > Ok, I'm probably doing something wrong here, but every time I try to
>>>> load a
>>>> > combined rule file with all of the VRT and ET rules enabled (~30K
>>>> rules) it
>>>> > fails following stage 3:
>>>> >
>>>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info>
>>>> (SigLoadSignatures) --
>>>> > 102 rule files processed. 30183 rules succesfully loaded, 164 rules
>>>> failed
>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>>>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800 are
>>>> IP-only
>>>> > rules, 20152 are inspecting packet payload, 11088 inspect application
>>>> layer,
>>>> > 0 are decoder event only
>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>>>> > (SigAddressPrepareStage1) -- building signature grouping structure,
>>>> stage 1:
>>>> > adding signatures to signature source addresses... complete
>>>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>>>> > (SigAddressPrepareStage2) -- building signature grouping structure,
>>>> stage 2:
>>>> > building source address list... complete
>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>>>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic
>>>> 1801173581, ctxs
>>>> > 0, avg per ctx 0)
>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>>>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>>>> > (SigAddressPrepareStage3) -- building signature grouping structure,
>>>> stage 3:
>>>> > building destination address lists... complete
>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error>
>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] -
>>>> SCMalloc
>>>> > failed: Cannot allocate memory, while trying to allocate 558852 bytes
>>>> >
>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583) <Error>
>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] - Out of
>>>> > memory. The engine cannot be initialized. Exiting...
>>>> >
>>>> > I have done this while watching the memory useage in top (set to
>>>> refresh
>>>> > every .2 seconds). Initially when this happened I only had 4GB
>>>> allocated to
>>>> > the VM. Useage never gets beyond 2GB so that left almost 2GB
>>>> available. I
>>>> > decided to bump the VM up to 8GB but the problem didn't go away. It
>>>> still
>>>> > exits when the memory useage gets to around 2GB.
>>>> >
>>>> > Everything works fine when I load a reduced ruleset, i.e. just VRT or
>>>> just
>>>> > ET, but for my tests I want to load both. Before I go back to the VM
>>>> > administrator and ask for 16 GB (and wait several days for the
>>>> allocation) I
>>>> > was wondering if there might be a config setting that is limiting the
>>>> size
>>>> > of memory allocated to the rules.
>>>> >
>>>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>>>> >
>>>> > Any suggestions are welcome.
>>>> >
>>>> > Gene
>>>> >
>>>> > --
>>>> > Gene Albin
>>>> > gene.albin at gmail.com
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Oisf-users mailing list
>>>> > Oisf-users at openinfosecfoundation.org
>>>> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>> >
>>>> >
>>>> _______________________________________________
>>>> Oisf-users mailing list
>>>> Oisf-users at openinfosecfoundation.org
>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>
>>>
>>>
>>>
>>> --
>>> "Of course, someone who knows more about this will correct me if I'm
>>> wrong, and someone who knows less will correct me if I'm right."
>>> David Palmer (palmer at tybalt.caltech.edu)
>>>
>>>
>>
>>
>> --
>> Gene Albin
>> gene.albin at gmail.com
>>
>>
>
>
> --
> "Of course, someone who knows more about this will correct me if I'm
> wrong, and someone who knows less will correct me if I'm right."
> David Palmer (palmer at tybalt.caltech.edu)
>
>
> _______________________________________________
> Oisf-users mailing listOisf-users at openinfosecfoundation.orghttp://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> In that respect.... What is your output of
> ulimit -aH
> and
> ulimit -a
> for the user that you run Suricata with?
>
> --
> Regards,
> Peter Manev
>
>
--
Gene Albin
gene.albin at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110728/aafd7acd/attachment-0002.html>
More information about the Oisf-users
mailing list