[Oisf-users] Suricata runs out of memory on startup
Gene Albin
gene.albin at gmail.com
Mon Aug 1 16:15:45 UTC 2011
Dave,
Peter is spot on with the 1G/3G thing. I'd love to move to a 64bit OS but
my project is running way short on time so I don't have time to build a new
64 bit VM and get it configured, up and running in time. My work around is
to use a subset of rules in order to keep it under the 3G limit. So far so
good. I cut out lots of rules by using the 'emerging-all.rules' instead of
loading each 'emerging-*.rules' file. That combined with the full set of
VRT rules allows me to stay under 3GB. While it's not what I wanted it
works and will have to do.
Peter, you mentioned valgrind. I just looked it upon the web and it looks a
bit beyond my novice programming skill. Is it painful to setup and use?
Gene
On Sun, Jul 31, 2011 at 8:36 AM, Dave Remien <dave.remien at gmail.com> wrote:
> Peter,
>
> Gene has a 1G/3G kernel and can indeed get 3G; I sent him a quickie prog
> that shows that. After that, seems like valgrind is in order to try to
> figure out where the extra memory is being allocated, if you can't see why
> from the suri.yaml files.... possibly an area Victor could shed light on.
> Or, as you say, move to a 64 bit kernel. At least there Gene can get 4G -
> 8-).
>
> Cheers,
>
> Dave
>
>
> On Sun, Jul 31, 2011 at 5:09 AM, Peter Manev <petermanev at gmail.com> wrote:
>
>>
>>
>> On Sat, Jul 30, 2011 at 5:49 PM, Peter Manev <petermanev at gmail.com>wrote:
>>
>>>
>>>
>>> On Fri, Jul 29, 2011 at 5:34 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>
>>>> Peter,
>>>> Sudo suricata ends with the same problem. I'm wondering if there may
>>>> be a difference between the optimization of our suricata.yaml files. Would
>>>> you mind exchanging files so that we can compare?
>>>>
>>>> Gene
>>>>
>>>>
>>>> On Fri, Jul 29, 2011 at 1:05 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Jul 28, 2011 at 8:59 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>
>>>>>> Gents,
>>>>>> Thanks all for the suggestions. Specific responses below
>>>>>>
>>>>>> Rmkml - I made your recommended change to the suricata.yaml file (-
>>>>>> profile: low) and there was no change. Watching top the spike appeard to
>>>>>> peak at 3.7GB then Suricata exited.
>>>>>>
>>>>>> Dave - I have to admit that I'm not versed in C so I can't build the
>>>>>> program you're describing. Do you know where I could find a pre-built one?
>>>>>> I agree that upgrading to a 64-bit version would be the best choice, however
>>>>>> I'm up against a very short timeline here and won't have time to upgrade the
>>>>>> OS the reinstall Suricata and the rest of the tools that I need.
>>>>>>
>>>>>> Peter - Here are the results from both ulimit -aH and ulimit -a:
>>>>>>
>>>>>> [gene at suri2 ~]$ ulimit -aH
>>>>>> core file size (blocks, -c) unlimited
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) 278528
>>>>>> max locked memory (kbytes, -l) 32
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 1024
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) unlimited
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) 278528
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>> [gene at suri2 ~]$ ulimit -a
>>>>>> core file size (blocks, -c) 0
>>>>>> data seg size (kbytes, -d) unlimited
>>>>>> scheduling priority (-e) 0
>>>>>> file size (blocks, -f) unlimited
>>>>>> pending signals (-i) 278528
>>>>>> max locked memory (kbytes, -l) 32
>>>>>> max memory size (kbytes, -m) unlimited
>>>>>> open files (-n) 1024
>>>>>> pipe size (512 bytes, -p) 8
>>>>>> POSIX message queues (bytes, -q) 819200
>>>>>> real-time priority (-r) 0
>>>>>> stack size (kbytes, -s) 10240
>>>>>> cpu time (seconds, -t) unlimited
>>>>>> max user processes (-u) 278528
>>>>>> virtual memory (kbytes, -v) unlimited
>>>>>> file locks (-x) unlimited
>>>>>>
>>>>>> I'm not exactly sure what I'm looking at here, but it looks like my
>>>>>> max memory size is unlimited. Am I reading this correctly?
>>>>>> I've also looked over your screen capture from your Ubuntu 32-bit VM
>>>>>> and can't figure out why I'm crashing. What version of Suricata are you
>>>>>> running on that 32-bit Ubuntu VM? 1.1b2?
>>>>>>
>>>>>> Gene
>>>>>>
>>>>>>
>>>>>> On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>>>
>>>>>>> **
>>>>>>> On 07/28/2011 02:54 PM, Dave Remien wrote:
>>>>>>>
>>>>>>> If you're up for it, about 15 lines of C code will give you a tiny
>>>>>>> program to test how much memory you can get for a single process - basically
>>>>>>> just malloc in a loop until you can't anymore. Sounds like your environment
>>>>>>> may actually be limited to 2GB of process size; normal for Linux is 3GB (all
>>>>>>> in the 32 bit world). Or you could lobby for a 64 bit copy
>>>>>>> of Centos; that'll eliminate the cap (for this purpose).
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Dave
>>>>>>>
>>>>>>> On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>>>
>>>>>>>> I just created a ticket with the details. To answer the questions
>>>>>>>> here, I'm running the 1.1b2 build from the tarball. Not using git. The
>>>>>>>> machine is running the 32 bit version of CentOS5.6, but we just applied the
>>>>>>>> kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is
>>>>>>>> this what you are talking about, Dave? Lastly I included the suricata.yaml
>>>>>>>> file as well as the output from free -m and my collectl memory statistics
>>>>>>>> during the fatal run.
>>>>>>>>
>>>>>>>> Thanks for helping out with this. I thought that bumping the ram up
>>>>>>>> to 16GB would fix it, but it appears not. Maybe I'll start slicing off some
>>>>>>>> rules and see where the threshold lies...
>>>>>>>>
>>>>>>>> Gene
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <
>>>>>>>> dave.remien at gmail.com> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <
>>>>>>>>> william.metcalf at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Can you create a redmine ticket and attach a scrubbed version of
>>>>>>>>>> your
>>>>>>>>>> suricata.yaml? Along with output of free -m prior to starting
>>>>>>>>>> suri?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Are you running a 32 bit kernel with a 2GB/2GB memory split, by
>>>>>>>>> any chance??
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Dave
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Will
>>>>>>>>>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <
>>>>>>>>>> gene.albin at gmail.com> wrote:
>>>>>>>>>> > Ok, I'm probably doing something wrong here, but every time I
>>>>>>>>>> try to load a
>>>>>>>>>> > combined rule file with all of the VRT and ET rules enabled
>>>>>>>>>> (~30K rules) it
>>>>>>>>>> > fails following stage 3:
>>>>>>>>>> >
>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info>
>>>>>>>>>> (SigLoadSignatures) --
>>>>>>>>>> > 102 rule files processed. 30183 rules succesfully loaded, 164
>>>>>>>>>> rules failed
>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>>>>>>>>>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800
>>>>>>>>>> are IP-only
>>>>>>>>>> > rules, 20152 are inspecting packet payload, 11088 inspect
>>>>>>>>>> application layer,
>>>>>>>>>> > 0 are decoder event only
>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>>>>>>>>>> > (SigAddressPrepareStage1) -- building signature grouping
>>>>>>>>>> structure, stage 1:
>>>>>>>>>> > adding signatures to signature source addresses... complete
>>>>>>>>>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>>>>>>>>>> > (SigAddressPrepareStage2) -- building signature grouping
>>>>>>>>>> structure, stage 2:
>>>>>>>>>> > building source address list... complete
>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>>>>>>>>>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic
>>>>>>>>>> 1801173581, ctxs
>>>>>>>>>> > 0, avg per ctx 0)
>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>>>>>>>>>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>>>>>>>>>> > (SigAddressPrepareStage3) -- building signature grouping
>>>>>>>>>> structure, stage 3:
>>>>>>>>>> > building destination address lists... complete
>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>>> <Error>
>>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] -
>>>>>>>>>> SCMalloc
>>>>>>>>>> > failed: Cannot allocate memory, while trying to allocate 558852
>>>>>>>>>> bytes
>>>>>>>>>> >
>>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>>> <Error>
>>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] -
>>>>>>>>>> Out of
>>>>>>>>>> > memory. The engine cannot be initialized. Exiting...
>>>>>>>>>> >
>>>>>>>>>> > I have done this while watching the memory useage in top (set to
>>>>>>>>>> refresh
>>>>>>>>>> > every .2 seconds). Initially when this happened I only had 4GB
>>>>>>>>>> allocated to
>>>>>>>>>> > the VM. Useage never gets beyond 2GB so that left almost 2GB
>>>>>>>>>> available. I
>>>>>>>>>> > decided to bump the VM up to 8GB but the problem didn't go
>>>>>>>>>> away. It still
>>>>>>>>>> > exits when the memory useage gets to around 2GB.
>>>>>>>>>> >
>>>>>>>>>> > Everything works fine when I load a reduced ruleset, i.e. just
>>>>>>>>>> VRT or just
>>>>>>>>>> > ET, but for my tests I want to load both. Before I go back to
>>>>>>>>>> the VM
>>>>>>>>>> > administrator and ask for 16 GB (and wait several days for the
>>>>>>>>>> allocation) I
>>>>>>>>>> > was wondering if there might be a config setting that is
>>>>>>>>>> limiting the size
>>>>>>>>>> > of memory allocated to the rules.
>>>>>>>>>> >
>>>>>>>>>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>>>>>>>>>> >
>>>>>>>>>> > Any suggestions are welcome.
>>>>>>>>>> >
>>>>>>>>>> > Gene
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > Gene Albin
>>>>>>>>>> > gene.albin at gmail.com
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > _______________________________________________
>>>>>>>>>> > Oisf-users mailing list
>>>>>>>>>> > Oisf-users at openinfosecfoundation.org
>>>>>>>>>> >
>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Oisf-users mailing list
>>>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> "Of course, someone who knows more about this will correct me if
>>>>>>>>> I'm
>>>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gene Albin
>>>>>>>> gene.albin at gmail.com
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> "Of course, someone who knows more about this will correct me if I'm
>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Oisf-users mailing listOisf-users at openinfosecfoundation.orghttp://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>
>>>>>>> In that respect.... What is your output of
>>>>>>> ulimit -aH
>>>>>>> and
>>>>>>> ulimit -a
>>>>>>> for the user that you run Suricata with?
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Peter Manev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gene Albin
>>>>>> gene.albin at gmail.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> Hi Gene,
>>>>> I am using Suricata version 1.1beta2 (rev df3ca32).
>>>>> What happens if you try to run Suricata as root?
>>>>>
>>>>> Thanks
>>>>>
>>>>>
>>>>> --
>>>>> Peter Manev
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Gene Albin
>>>> gene.albin at gmail.com
>>>>
>>>>
>>>
>>>
>>> Sure,
>>> I did send mine to you.
>>> I tried with yours and Suricata is running aswell (I only modified the
>>> rules directory)
>>>
>>> Thanks
>>>
>>> --
>>> Peter Manev
>>>
>>
>> Hi Gene
>>
>> I think Dave is right - you might be running 2Gb/2Gb memory split on a
>> 32bit.
>> I am running 1G / 3G memory split which gives me the edge I believe.
>> I am inclined to believe(please correct me if I am wrong ) that you have a
>> 2Gb per process limit because of that (since you said yourself that Suricata
>> always gets killed when it reaches 2Gb of use at start up ) the PAE is not
>> going to help you in this case - if you would like to check that for sure
>> you could try 64bit system.
>>
>> Thanks
>> --
>> Peter Manev
>>
>
>
>
> --
> "Of course, someone who knows more about this will correct me if I'm
> wrong, and someone who knows less will correct me if I'm right."
> David Palmer (palmer at tybalt.caltech.edu)
>
>
--
Gene Albin
gene.albin at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110801/44a60d00/attachment-0002.html>
More information about the Oisf-users
mailing list