[Oisf-users] Suricata runs out of memory on startup
Peter Manev
petermanev at gmail.com
Mon Aug 1 16:26:05 UTC 2011
On Mon, Aug 1, 2011 at 6:15 PM, Gene Albin <gene.albin at gmail.com> wrote:
> Dave,
> Peter is spot on with the 1G/3G thing. I'd love to move to a 64bit OS
> but my project is running way short on time so I don't have time to build a
> new 64 bit VM and get it configured, up and running in time. My work around
> is to use a subset of rules in order to keep it under the 3G limit. So far
> so good. I cut out lots of rules by using the 'emerging-all.rules' instead
> of loading each 'emerging-*.rules' file. That combined with the full set of
> VRT rules allows me to stay under 3GB. While it's not what I wanted it
> works and will have to do.
>
> Peter, you mentioned valgrind. I just looked it upon the web and it looks
> a bit beyond my novice programming skill. Is it painful to setup and use?
>
> Gene
>
>
> On Sun, Jul 31, 2011 at 8:36 AM, Dave Remien <dave.remien at gmail.com>wrote:
>
>> Peter,
>>
>> Gene has a 1G/3G kernel and can indeed get 3G; I sent him a quickie prog
>> that shows that. After that, seems like valgrind is in order to try to
>> figure out where the extra memory is being allocated, if you can't see why
>> from the suri.yaml files.... possibly an area Victor could shed light on.
>> Or, as you say, move to a 64 bit kernel. At least there Gene can get 4G -
>> 8-).
>>
>> Cheers,
>>
>> Dave
>>
>>
>> On Sun, Jul 31, 2011 at 5:09 AM, Peter Manev <petermanev at gmail.com>wrote:
>>
>>>
>>>
>>> On Sat, Jul 30, 2011 at 5:49 PM, Peter Manev <petermanev at gmail.com>wrote:
>>>
>>>>
>>>>
>>>> On Fri, Jul 29, 2011 at 5:34 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>
>>>>> Peter,
>>>>> Sudo suricata ends with the same problem. I'm wondering if there may
>>>>> be a difference between the optimization of our suricata.yaml files. Would
>>>>> you mind exchanging files so that we can compare?
>>>>>
>>>>> Gene
>>>>>
>>>>>
>>>>> On Fri, Jul 29, 2011 at 1:05 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 28, 2011 at 8:59 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>>
>>>>>>> Gents,
>>>>>>> Thanks all for the suggestions. Specific responses below
>>>>>>>
>>>>>>> Rmkml - I made your recommended change to the suricata.yaml file (-
>>>>>>> profile: low) and there was no change. Watching top the spike appeard to
>>>>>>> peak at 3.7GB then Suricata exited.
>>>>>>>
>>>>>>> Dave - I have to admit that I'm not versed in C so I can't build the
>>>>>>> program you're describing. Do you know where I could find a pre-built one?
>>>>>>> I agree that upgrading to a 64-bit version would be the best choice, however
>>>>>>> I'm up against a very short timeline here and won't have time to upgrade the
>>>>>>> OS the reinstall Suricata and the rest of the tools that I need.
>>>>>>>
>>>>>>> Peter - Here are the results from both ulimit -aH and ulimit -a:
>>>>>>>
>>>>>>> [gene at suri2 ~]$ ulimit -aH
>>>>>>> core file size (blocks, -c) unlimited
>>>>>>> data seg size (kbytes, -d) unlimited
>>>>>>> scheduling priority (-e) 0
>>>>>>> file size (blocks, -f) unlimited
>>>>>>> pending signals (-i) 278528
>>>>>>> max locked memory (kbytes, -l) 32
>>>>>>> max memory size (kbytes, -m) unlimited
>>>>>>> open files (-n) 1024
>>>>>>> pipe size (512 bytes, -p) 8
>>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>>> real-time priority (-r) 0
>>>>>>> stack size (kbytes, -s) unlimited
>>>>>>> cpu time (seconds, -t) unlimited
>>>>>>> max user processes (-u) 278528
>>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>>> file locks (-x) unlimited
>>>>>>>
>>>>>>> [gene at suri2 ~]$ ulimit -a
>>>>>>> core file size (blocks, -c) 0
>>>>>>> data seg size (kbytes, -d) unlimited
>>>>>>> scheduling priority (-e) 0
>>>>>>> file size (blocks, -f) unlimited
>>>>>>> pending signals (-i) 278528
>>>>>>> max locked memory (kbytes, -l) 32
>>>>>>> max memory size (kbytes, -m) unlimited
>>>>>>> open files (-n) 1024
>>>>>>> pipe size (512 bytes, -p) 8
>>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>>> real-time priority (-r) 0
>>>>>>> stack size (kbytes, -s) 10240
>>>>>>> cpu time (seconds, -t) unlimited
>>>>>>> max user processes (-u) 278528
>>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>>> file locks (-x) unlimited
>>>>>>>
>>>>>>> I'm not exactly sure what I'm looking at here, but it looks like my
>>>>>>> max memory size is unlimited. Am I reading this correctly?
>>>>>>> I've also looked over your screen capture from your Ubuntu 32-bit VM
>>>>>>> and can't figure out why I'm crashing. What version of Suricata are you
>>>>>>> running on that 32-bit Ubuntu VM? 1.1b2?
>>>>>>>
>>>>>>> Gene
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>>>>
>>>>>>>> **
>>>>>>>> On 07/28/2011 02:54 PM, Dave Remien wrote:
>>>>>>>>
>>>>>>>> If you're up for it, about 15 lines of C code will give you a tiny
>>>>>>>> program to test how much memory you can get for a single process - basically
>>>>>>>> just malloc in a loop until you can't anymore. Sounds like your environment
>>>>>>>> may actually be limited to 2GB of process size; normal for Linux is 3GB (all
>>>>>>>> in the 32 bit world). Or you could lobby for a 64 bit copy
>>>>>>>> of Centos; that'll eliminate the cap (for this purpose).
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Dave
>>>>>>>>
>>>>>>>> On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>>>>
>>>>>>>>> I just created a ticket with the details. To answer the questions
>>>>>>>>> here, I'm running the 1.1b2 build from the tarball. Not using git. The
>>>>>>>>> machine is running the 32 bit version of CentOS5.6, but we just applied the
>>>>>>>>> kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is
>>>>>>>>> this what you are talking about, Dave? Lastly I included the suricata.yaml
>>>>>>>>> file as well as the output from free -m and my collectl memory statistics
>>>>>>>>> during the fatal run.
>>>>>>>>>
>>>>>>>>> Thanks for helping out with this. I thought that bumping the ram
>>>>>>>>> up to 16GB would fix it, but it appears not. Maybe I'll start slicing off
>>>>>>>>> some rules and see where the threshold lies...
>>>>>>>>>
>>>>>>>>> Gene
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <
>>>>>>>>> dave.remien at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <
>>>>>>>>>> william.metcalf at gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Can you create a redmine ticket and attach a scrubbed version of
>>>>>>>>>>> your
>>>>>>>>>>> suricata.yaml? Along with output of free -m prior to starting
>>>>>>>>>>> suri?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Are you running a 32 bit kernel with a 2GB/2GB memory split, by
>>>>>>>>>> any chance??
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Dave
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata
>>>>>>>>>>>
>>>>>>>>>>> Regards,
>>>>>>>>>>>
>>>>>>>>>>> Will
>>>>>>>>>>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <
>>>>>>>>>>> gene.albin at gmail.com> wrote:
>>>>>>>>>>> > Ok, I'm probably doing something wrong here, but every time I
>>>>>>>>>>> try to load a
>>>>>>>>>>> > combined rule file with all of the VRT and ET rules enabled
>>>>>>>>>>> (~30K rules) it
>>>>>>>>>>> > fails following stage 3:
>>>>>>>>>>> >
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info>
>>>>>>>>>>> (SigLoadSignatures) --
>>>>>>>>>>> > 102 rule files processed. 30183 rules succesfully loaded, 164
>>>>>>>>>>> rules failed
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>>>>>>>>>>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800
>>>>>>>>>>> are IP-only
>>>>>>>>>>> > rules, 20152 are inspecting packet payload, 11088 inspect
>>>>>>>>>>> application layer,
>>>>>>>>>>> > 0 are decoder event only
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>>>>>>>>>>> > (SigAddressPrepareStage1) -- building signature grouping
>>>>>>>>>>> structure, stage 1:
>>>>>>>>>>> > adding signatures to signature source addresses... complete
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>>>>>>>>>>> > (SigAddressPrepareStage2) -- building signature grouping
>>>>>>>>>>> structure, stage 2:
>>>>>>>>>>> > building source address list... complete
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>>>>>>>>>>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic
>>>>>>>>>>> 1801173581, ctxs
>>>>>>>>>>> > 0, avg per ctx 0)
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>>>>>>>>>>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>>>>>>>>>>> > (SigAddressPrepareStage3) -- building signature grouping
>>>>>>>>>>> structure, stage 3:
>>>>>>>>>>> > building destination address lists... complete
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>>>> <Error>
>>>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)]
>>>>>>>>>>> - SCMalloc
>>>>>>>>>>> > failed: Cannot allocate memory, while trying to allocate 558852
>>>>>>>>>>> bytes
>>>>>>>>>>> >
>>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>>>> <Error>
>>>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] -
>>>>>>>>>>> Out of
>>>>>>>>>>> > memory. The engine cannot be initialized. Exiting...
>>>>>>>>>>> >
>>>>>>>>>>> > I have done this while watching the memory useage in top (set
>>>>>>>>>>> to refresh
>>>>>>>>>>> > every .2 seconds). Initially when this happened I only had 4GB
>>>>>>>>>>> allocated to
>>>>>>>>>>> > the VM. Useage never gets beyond 2GB so that left almost 2GB
>>>>>>>>>>> available. I
>>>>>>>>>>> > decided to bump the VM up to 8GB but the problem didn't go
>>>>>>>>>>> away. It still
>>>>>>>>>>> > exits when the memory useage gets to around 2GB.
>>>>>>>>>>> >
>>>>>>>>>>> > Everything works fine when I load a reduced ruleset, i.e. just
>>>>>>>>>>> VRT or just
>>>>>>>>>>> > ET, but for my tests I want to load both. Before I go back to
>>>>>>>>>>> the VM
>>>>>>>>>>> > administrator and ask for 16 GB (and wait several days for the
>>>>>>>>>>> allocation) I
>>>>>>>>>>> > was wondering if there might be a config setting that is
>>>>>>>>>>> limiting the size
>>>>>>>>>>> > of memory allocated to the rules.
>>>>>>>>>>> >
>>>>>>>>>>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>>>>>>>>>>> >
>>>>>>>>>>> > Any suggestions are welcome.
>>>>>>>>>>> >
>>>>>>>>>>> > Gene
>>>>>>>>>>> >
>>>>>>>>>>> > --
>>>>>>>>>>> > Gene Albin
>>>>>>>>>>> > gene.albin at gmail.com
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> > _______________________________________________
>>>>>>>>>>> > Oisf-users mailing list
>>>>>>>>>>> > Oisf-users at openinfosecfoundation.org
>>>>>>>>>>> >
>>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>>> >
>>>>>>>>>>> >
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Oisf-users mailing list
>>>>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>>>>>
>>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> "Of course, someone who knows more about this will correct me if
>>>>>>>>>> I'm
>>>>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Gene Albin
>>>>>>>>> gene.albin at gmail.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> "Of course, someone who knows more about this will correct me if I'm
>>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-users mailing listOisf-users at openinfosecfoundation.orghttp://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>
>>>>>>>> In that respect.... What is your output of
>>>>>>>> ulimit -aH
>>>>>>>> and
>>>>>>>> ulimit -a
>>>>>>>> for the user that you run Suricata with?
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Peter Manev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gene Albin
>>>>>>> gene.albin at gmail.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi Gene,
>>>>>> I am using Suricata version 1.1beta2 (rev df3ca32).
>>>>>> What happens if you try to run Suricata as root?
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Peter Manev
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gene Albin
>>>>> gene.albin at gmail.com
>>>>>
>>>>>
>>>>
>>>>
>>>> Sure,
>>>> I did send mine to you.
>>>> I tried with yours and Suricata is running aswell (I only modified the
>>>> rules directory)
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> Peter Manev
>>>>
>>>
>>> Hi Gene
>>>
>>> I think Dave is right - you might be running 2Gb/2Gb memory split on a
>>> 32bit.
>>> I am running 1G / 3G memory split which gives me the edge I believe.
>>> I am inclined to believe(please correct me if I am wrong ) that you have
>>> a 2Gb per process limit because of that (since you said yourself that
>>> Suricata always gets killed when it reaches 2Gb of use at start up ) the PAE
>>> is not going to help you in this case - if you would like to check that for
>>> sure you could try 64bit system.
>>>
>>> Thanks
>>> --
>>> Peter Manev
>>>
>>
>>
>>
>> --
>> "Of course, someone who knows more about this will correct me if I'm
>> wrong, and someone who knows less will correct me if I'm right."
>> David Palmer (palmer at tybalt.caltech.edu)
>>
>>
>
>
> --
> Gene Albin
> gene.albin at gmail.com
>
>
"Peter, you mentioned valgrind. I just looked it upon the web and it looks
a bit beyond my novice programming skill. Is it painful to setup and use?"
- Dave mentioned that.
--
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110801/5dc5acd2/attachment-0002.html>
More information about the Oisf-users
mailing list