[Oisf-users] Suricata runs out of memory on startup
Dave Remien
dave.remien at gmail.com
Wed Aug 3 01:52:13 UTC 2011
Not particularly. Probably installable from the package manager in your
Linux distro.
A starting point would be to run the leak-checker:
valgrind --leak-check=yes suricata [suri args]...
It'll probably fail outright, since it causes extra memory to be alloc'd,
and that's already your problem 8-/. You might be able to set down some of
the things in the sur.yaml file that chew up memory, and leave some of the
rules out; that way you might be able to see what's hogging memory.
Things can get more exotic fairly quickly from here...
Cheers,
Dave
On Mon, Aug 1, 2011 at 10:26 AM, Peter Manev <petermanev at gmail.com> wrote:
>
>
> On Mon, Aug 1, 2011 at 6:15 PM, Gene Albin <gene.albin at gmail.com> wrote:
>
>> Dave,
>> Peter is spot on with the 1G/3G thing. I'd love to move to a 64bit OS
>> but my project is running way short on time so I don't have time to build a
>> new 64 bit VM and get it configured, up and running in time. My work around
>> is to use a subset of rules in order to keep it under the 3G limit. So far
>> so good. I cut out lots of rules by using the 'emerging-all.rules' instead
>> of loading each 'emerging-*.rules' file. That combined with the full set of
>> VRT rules allows me to stay under 3GB. While it's not what I wanted it
>> works and will have to do.
>>
>> Peter, you mentioned valgrind. I just looked it upon the web and it looks
>> a bit beyond my novice programming skill. Is it painful to setup and use?
>>
>> Gene
>>
>>
>> On Sun, Jul 31, 2011 at 8:36 AM, Dave Remien <dave.remien at gmail.com>wrote:
>>
>>> Peter,
>>>
>>> Gene has a 1G/3G kernel and can indeed get 3G; I sent him a quickie prog
>>> that shows that. After that, seems like valgrind is in order to try to
>>> figure out where the extra memory is being allocated, if you can't see why
>>> from the suri.yaml files.... possibly an area Victor could shed light on.
>>> Or, as you say, move to a 64 bit kernel. At least there Gene can get 4G -
>>> 8-).
>>>
>>> Cheers,
>>>
>>> Dave
>>>
>>>
>>> On Sun, Jul 31, 2011 at 5:09 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>
>>>>
>>>>
>>>> On Sat, Jul 30, 2011 at 5:49 PM, Peter Manev <petermanev at gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 29, 2011 at 5:34 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>
>>>>>> Peter,
>>>>>> Sudo suricata ends with the same problem. I'm wondering if there
>>>>>> may be a difference between the optimization of our suricata.yaml files.
>>>>>> Would you mind exchanging files so that we can compare?
>>>>>>
>>>>>> Gene
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 29, 2011 at 1:05 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 28, 2011 at 8:59 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>>>
>>>>>>>> Gents,
>>>>>>>> Thanks all for the suggestions. Specific responses below
>>>>>>>>
>>>>>>>> Rmkml - I made your recommended change to the suricata.yaml file (-
>>>>>>>> profile: low) and there was no change. Watching top the spike appeard to
>>>>>>>> peak at 3.7GB then Suricata exited.
>>>>>>>>
>>>>>>>> Dave - I have to admit that I'm not versed in C so I can't build the
>>>>>>>> program you're describing. Do you know where I could find a pre-built one?
>>>>>>>> I agree that upgrading to a 64-bit version would be the best choice, however
>>>>>>>> I'm up against a very short timeline here and won't have time to upgrade the
>>>>>>>> OS the reinstall Suricata and the rest of the tools that I need.
>>>>>>>>
>>>>>>>> Peter - Here are the results from both ulimit -aH and ulimit -a:
>>>>>>>>
>>>>>>>> [gene at suri2 ~]$ ulimit -aH
>>>>>>>> core file size (blocks, -c) unlimited
>>>>>>>> data seg size (kbytes, -d) unlimited
>>>>>>>> scheduling priority (-e) 0
>>>>>>>> file size (blocks, -f) unlimited
>>>>>>>> pending signals (-i) 278528
>>>>>>>> max locked memory (kbytes, -l) 32
>>>>>>>> max memory size (kbytes, -m) unlimited
>>>>>>>> open files (-n) 1024
>>>>>>>> pipe size (512 bytes, -p) 8
>>>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>>>> real-time priority (-r) 0
>>>>>>>> stack size (kbytes, -s) unlimited
>>>>>>>> cpu time (seconds, -t) unlimited
>>>>>>>> max user processes (-u) 278528
>>>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>>>> file locks (-x) unlimited
>>>>>>>>
>>>>>>>> [gene at suri2 ~]$ ulimit -a
>>>>>>>> core file size (blocks, -c) 0
>>>>>>>> data seg size (kbytes, -d) unlimited
>>>>>>>> scheduling priority (-e) 0
>>>>>>>> file size (blocks, -f) unlimited
>>>>>>>> pending signals (-i) 278528
>>>>>>>> max locked memory (kbytes, -l) 32
>>>>>>>> max memory size (kbytes, -m) unlimited
>>>>>>>> open files (-n) 1024
>>>>>>>> pipe size (512 bytes, -p) 8
>>>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>>>> real-time priority (-r) 0
>>>>>>>> stack size (kbytes, -s) 10240
>>>>>>>> cpu time (seconds, -t) unlimited
>>>>>>>> max user processes (-u) 278528
>>>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>>>> file locks (-x) unlimited
>>>>>>>>
>>>>>>>> I'm not exactly sure what I'm looking at here, but it looks like my
>>>>>>>> max memory size is unlimited. Am I reading this correctly?
>>>>>>>> I've also looked over your screen capture from your Ubuntu 32-bit VM
>>>>>>>> and can't figure out why I'm crashing. What version of Suricata are you
>>>>>>>> running on that 32-bit Ubuntu VM? 1.1b2?
>>>>>>>>
>>>>>>>> Gene
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> **
>>>>>>>>> On 07/28/2011 02:54 PM, Dave Remien wrote:
>>>>>>>>>
>>>>>>>>> If you're up for it, about 15 lines of C code will give you a tiny
>>>>>>>>> program to test how much memory you can get for a single process - basically
>>>>>>>>> just malloc in a loop until you can't anymore. Sounds like your environment
>>>>>>>>> may actually be limited to 2GB of process size; normal for Linux is 3GB (all
>>>>>>>>> in the 32 bit world). Or you could lobby for a 64 bit copy
>>>>>>>>> of Centos; that'll eliminate the cap (for this purpose).
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>> On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>>>>>
>>>>>>>>>> I just created a ticket with the details. To answer the questions
>>>>>>>>>> here, I'm running the 1.1b2 build from the tarball. Not using git. The
>>>>>>>>>> machine is running the 32 bit version of CentOS5.6, but we just applied the
>>>>>>>>>> kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is
>>>>>>>>>> this what you are talking about, Dave? Lastly I included the suricata.yaml
>>>>>>>>>> file as well as the output from free -m and my collectl memory statistics
>>>>>>>>>> during the fatal run.
>>>>>>>>>>
>>>>>>>>>> Thanks for helping out with this. I thought that bumping the ram
>>>>>>>>>> up to 16GB would fix it, but it appears not. Maybe I'll start slicing off
>>>>>>>>>> some rules and see where the threshold lies...
>>>>>>>>>>
>>>>>>>>>> Gene
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <
>>>>>>>>>> dave.remien at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <
>>>>>>>>>>> william.metcalf at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Can you create a redmine ticket and attach a scrubbed version of
>>>>>>>>>>>> your
>>>>>>>>>>>> suricata.yaml? Along with output of free -m prior to starting
>>>>>>>>>>>> suri?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Are you running a 32 bit kernel with a 2GB/2GB memory split, by
>>>>>>>>>>> any chance??
>>>>>>>>>>>
>>>>>>>>>>> Cheers,
>>>>>>>>>>>
>>>>>>>>>>> Dave
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata
>>>>>>>>>>>>
>>>>>>>>>>>> Regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Will
>>>>>>>>>>>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <
>>>>>>>>>>>> gene.albin at gmail.com> wrote:
>>>>>>>>>>>> > Ok, I'm probably doing something wrong here, but every time I
>>>>>>>>>>>> try to load a
>>>>>>>>>>>> > combined rule file with all of the VRT and ET rules enabled
>>>>>>>>>>>> (~30K rules) it
>>>>>>>>>>>> > fails following stage 3:
>>>>>>>>>>>> >
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info>
>>>>>>>>>>>> (SigLoadSignatures) --
>>>>>>>>>>>> > 102 rule files processed. 30183 rules succesfully loaded, 164
>>>>>>>>>>>> rules failed
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>>>>>>>>>>>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800
>>>>>>>>>>>> are IP-only
>>>>>>>>>>>> > rules, 20152 are inspecting packet payload, 11088 inspect
>>>>>>>>>>>> application layer,
>>>>>>>>>>>> > 0 are decoder event only
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>>>>>>>>>>>> > (SigAddressPrepareStage1) -- building signature grouping
>>>>>>>>>>>> structure, stage 1:
>>>>>>>>>>>> > adding signatures to signature source addresses... complete
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>>>>>>>>>>>> > (SigAddressPrepareStage2) -- building signature grouping
>>>>>>>>>>>> structure, stage 2:
>>>>>>>>>>>> > building source address list... complete
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>>>>>>>>>>>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic
>>>>>>>>>>>> 1801173581, ctxs
>>>>>>>>>>>> > 0, avg per ctx 0)
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>>>>>>>>>>>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>>>>>>>>>>>> > (SigAddressPrepareStage3) -- building signature grouping
>>>>>>>>>>>> structure, stage 3:
>>>>>>>>>>>> > building destination address lists... complete
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>>>>> <Error>
>>>>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)]
>>>>>>>>>>>> - SCMalloc
>>>>>>>>>>>> > failed: Cannot allocate memory, while trying to allocate
>>>>>>>>>>>> 558852 bytes
>>>>>>>>>>>> >
>>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>>>>> <Error>
>>>>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] -
>>>>>>>>>>>> Out of
>>>>>>>>>>>> > memory. The engine cannot be initialized. Exiting...
>>>>>>>>>>>> >
>>>>>>>>>>>> > I have done this while watching the memory useage in top (set
>>>>>>>>>>>> to refresh
>>>>>>>>>>>> > every .2 seconds). Initially when this happened I only had
>>>>>>>>>>>> 4GB allocated to
>>>>>>>>>>>> > the VM. Useage never gets beyond 2GB so that left almost 2GB
>>>>>>>>>>>> available. I
>>>>>>>>>>>> > decided to bump the VM up to 8GB but the problem didn't go
>>>>>>>>>>>> away. It still
>>>>>>>>>>>> > exits when the memory useage gets to around 2GB.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Everything works fine when I load a reduced ruleset, i.e. just
>>>>>>>>>>>> VRT or just
>>>>>>>>>>>> > ET, but for my tests I want to load both. Before I go back to
>>>>>>>>>>>> the VM
>>>>>>>>>>>> > administrator and ask for 16 GB (and wait several days for the
>>>>>>>>>>>> allocation) I
>>>>>>>>>>>> > was wondering if there might be a config setting that is
>>>>>>>>>>>> limiting the size
>>>>>>>>>>>> > of memory allocated to the rules.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Any suggestions are welcome.
>>>>>>>>>>>> >
>>>>>>>>>>>> > Gene
>>>>>>>>>>>> >
>>>>>>>>>>>> > --
>>>>>>>>>>>> > Gene Albin
>>>>>>>>>>>> > gene.albin at gmail.com
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>>> > Oisf-users mailing list
>>>>>>>>>>>> > Oisf-users at openinfosecfoundation.org
>>>>>>>>>>>> >
>>>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>>>> >
>>>>>>>>>>>> >
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Oisf-users mailing list
>>>>>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>>>>>>
>>>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> "Of course, someone who knows more about this will correct me if
>>>>>>>>>>> I'm
>>>>>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Gene Albin
>>>>>>>>>> gene.albin at gmail.com
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> "Of course, someone who knows more about this will correct me if
>>>>>>>>> I'm
>>>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Oisf-users mailing listOisf-users at openinfosecfoundation.orghttp://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>
>>>>>>>>> In that respect.... What is your output of
>>>>>>>>> ulimit -aH
>>>>>>>>> and
>>>>>>>>> ulimit -a
>>>>>>>>> for the user that you run Suricata with?
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Peter Manev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gene Albin
>>>>>>>> gene.albin at gmail.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hi Gene,
>>>>>>> I am using Suricata version 1.1beta2 (rev df3ca32).
>>>>>>> What happens if you try to run Suricata as root?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Peter Manev
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gene Albin
>>>>>> gene.albin at gmail.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Sure,
>>>>> I did send mine to you.
>>>>> I tried with yours and Suricata is running aswell (I only modified the
>>>>> rules directory)
>>>>>
>>>>> Thanks
>>>>>
>>>>> --
>>>>> Peter Manev
>>>>>
>>>>
>>>> Hi Gene
>>>>
>>>> I think Dave is right - you might be running 2Gb/2Gb memory split on a
>>>> 32bit.
>>>> I am running 1G / 3G memory split which gives me the edge I believe.
>>>> I am inclined to believe(please correct me if I am wrong ) that you have
>>>> a 2Gb per process limit because of that (since you said yourself that
>>>> Suricata always gets killed when it reaches 2Gb of use at start up ) the PAE
>>>> is not going to help you in this case - if you would like to check that for
>>>> sure you could try 64bit system.
>>>>
>>>> Thanks
>>>> --
>>>> Peter Manev
>>>>
>>>
>>>
>>>
>>> --
>>> "Of course, someone who knows more about this will correct me if I'm
>>> wrong, and someone who knows less will correct me if I'm right."
>>> David Palmer (palmer at tybalt.caltech.edu)
>>>
>>>
>>
>>
>> --
>> Gene Albin
>> gene.albin at gmail.com
>>
>>
>
> "Peter, you mentioned valgrind. I just looked it upon the web and it looks
> a bit beyond my novice programming skill. Is it painful to setup and use?"
> - Dave mentioned that.
>
>
> --
> Peter Manev
>
--
"Of course, someone who knows more about this will correct me if I'm
wrong, and someone who knows less will correct me if I'm right."
David Palmer (palmer at tybalt.caltech.edu)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110802/e6330c57/attachment-0002.html>
More information about the Oisf-users
mailing list