[Oisf-users] Suricata runs out of memory on startup
Peter Manev
petermanev at gmail.com
Sun Jul 31 11:09:05 UTC 2011
On Sat, Jul 30, 2011 at 5:49 PM, Peter Manev <petermanev at gmail.com> wrote:
>
>
> On Fri, Jul 29, 2011 at 5:34 PM, Gene Albin <gene.albin at gmail.com> wrote:
>
>> Peter,
>> Sudo suricata ends with the same problem. I'm wondering if there may be
>> a difference between the optimization of our suricata.yaml files. Would you
>> mind exchanging files so that we can compare?
>>
>> Gene
>>
>>
>> On Fri, Jul 29, 2011 at 1:05 AM, Peter Manev <petermanev at gmail.com>wrote:
>>
>>>
>>>
>>> On Thu, Jul 28, 2011 at 8:59 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>
>>>> Gents,
>>>> Thanks all for the suggestions. Specific responses below
>>>>
>>>> Rmkml - I made your recommended change to the suricata.yaml file (-
>>>> profile: low) and there was no change. Watching top the spike appeard to
>>>> peak at 3.7GB then Suricata exited.
>>>>
>>>> Dave - I have to admit that I'm not versed in C so I can't build the
>>>> program you're describing. Do you know where I could find a pre-built one?
>>>> I agree that upgrading to a 64-bit version would be the best choice, however
>>>> I'm up against a very short timeline here and won't have time to upgrade the
>>>> OS the reinstall Suricata and the rest of the tools that I need.
>>>>
>>>> Peter - Here are the results from both ulimit -aH and ulimit -a:
>>>>
>>>> [gene at suri2 ~]$ ulimit -aH
>>>> core file size (blocks, -c) unlimited
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 278528
>>>> max locked memory (kbytes, -l) 32
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 1024
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) unlimited
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 278528
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>> [gene at suri2 ~]$ ulimit -a
>>>> core file size (blocks, -c) 0
>>>> data seg size (kbytes, -d) unlimited
>>>> scheduling priority (-e) 0
>>>> file size (blocks, -f) unlimited
>>>> pending signals (-i) 278528
>>>> max locked memory (kbytes, -l) 32
>>>> max memory size (kbytes, -m) unlimited
>>>> open files (-n) 1024
>>>> pipe size (512 bytes, -p) 8
>>>> POSIX message queues (bytes, -q) 819200
>>>> real-time priority (-r) 0
>>>> stack size (kbytes, -s) 10240
>>>> cpu time (seconds, -t) unlimited
>>>> max user processes (-u) 278528
>>>> virtual memory (kbytes, -v) unlimited
>>>> file locks (-x) unlimited
>>>>
>>>> I'm not exactly sure what I'm looking at here, but it looks like my max
>>>> memory size is unlimited. Am I reading this correctly?
>>>> I've also looked over your screen capture from your Ubuntu 32-bit VM and
>>>> can't figure out why I'm crashing. What version of Suricata are you running
>>>> on that 32-bit Ubuntu VM? 1.1b2?
>>>>
>>>> Gene
>>>>
>>>>
>>>> On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>
>>>>> **
>>>>> On 07/28/2011 02:54 PM, Dave Remien wrote:
>>>>>
>>>>> If you're up for it, about 15 lines of C code will give you a tiny
>>>>> program to test how much memory you can get for a single process - basically
>>>>> just malloc in a loop until you can't anymore. Sounds like your environment
>>>>> may actually be limited to 2GB of process size; normal for Linux is 3GB (all
>>>>> in the 32 bit world). Or you could lobby for a 64 bit copy
>>>>> of Centos; that'll eliminate the cap (for this purpose).
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Dave
>>>>>
>>>>> On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>
>>>>>> I just created a ticket with the details. To answer the questions
>>>>>> here, I'm running the 1.1b2 build from the tarball. Not using git. The
>>>>>> machine is running the 32 bit version of CentOS5.6, but we just applied the
>>>>>> kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is
>>>>>> this what you are talking about, Dave? Lastly I included the suricata.yaml
>>>>>> file as well as the output from free -m and my collectl memory statistics
>>>>>> during the fatal run.
>>>>>>
>>>>>> Thanks for helping out with this. I thought that bumping the ram up
>>>>>> to 16GB would fix it, but it appears not. Maybe I'll start slicing off some
>>>>>> rules and see where the threshold lies...
>>>>>>
>>>>>> Gene
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <dave.remien at gmail.com>wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <
>>>>>>> william.metcalf at gmail.com> wrote:
>>>>>>>
>>>>>>>> Can you create a redmine ticket and attach a scrubbed version of
>>>>>>>> your
>>>>>>>> suricata.yaml? Along with output of free -m prior to starting suri?
>>>>>>>>
>>>>>>>
>>>>>>> Are you running a 32 bit kernel with a 2GB/2GB memory split, by any
>>>>>>> chance??
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Dave
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Will
>>>>>>>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <gene.albin at gmail.com>
>>>>>>>> wrote:
>>>>>>>> > Ok, I'm probably doing something wrong here, but every time I try
>>>>>>>> to load a
>>>>>>>> > combined rule file with all of the VRT and ET rules enabled (~30K
>>>>>>>> rules) it
>>>>>>>> > fails following stage 3:
>>>>>>>> >
>>>>>>>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info>
>>>>>>>> (SigLoadSignatures) --
>>>>>>>> > 102 rule files processed. 30183 rules succesfully loaded, 164
>>>>>>>> rules failed
>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>>>>>>>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800 are
>>>>>>>> IP-only
>>>>>>>> > rules, 20152 are inspecting packet payload, 11088 inspect
>>>>>>>> application layer,
>>>>>>>> > 0 are decoder event only
>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>>>>>>>> > (SigAddressPrepareStage1) -- building signature grouping
>>>>>>>> structure, stage 1:
>>>>>>>> > adding signatures to signature source addresses... complete
>>>>>>>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>>>>>>>> > (SigAddressPrepareStage2) -- building signature grouping
>>>>>>>> structure, stage 2:
>>>>>>>> > building source address list... complete
>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>>>>>>>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic
>>>>>>>> 1801173581, ctxs
>>>>>>>> > 0, avg per ctx 0)
>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>>>>>>>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>>>>>>>> > (SigAddressPrepareStage3) -- building signature grouping
>>>>>>>> structure, stage 3:
>>>>>>>> > building destination address lists... complete
>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>> <Error>
>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] -
>>>>>>>> SCMalloc
>>>>>>>> > failed: Cannot allocate memory, while trying to allocate 558852
>>>>>>>> bytes
>>>>>>>> >
>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>> <Error>
>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] - Out
>>>>>>>> of
>>>>>>>> > memory. The engine cannot be initialized. Exiting...
>>>>>>>> >
>>>>>>>> > I have done this while watching the memory useage in top (set to
>>>>>>>> refresh
>>>>>>>> > every .2 seconds). Initially when this happened I only had 4GB
>>>>>>>> allocated to
>>>>>>>> > the VM. Useage never gets beyond 2GB so that left almost 2GB
>>>>>>>> available. I
>>>>>>>> > decided to bump the VM up to 8GB but the problem didn't go away.
>>>>>>>> It still
>>>>>>>> > exits when the memory useage gets to around 2GB.
>>>>>>>> >
>>>>>>>> > Everything works fine when I load a reduced ruleset, i.e. just VRT
>>>>>>>> or just
>>>>>>>> > ET, but for my tests I want to load both. Before I go back to the
>>>>>>>> VM
>>>>>>>> > administrator and ask for 16 GB (and wait several days for the
>>>>>>>> allocation) I
>>>>>>>> > was wondering if there might be a config setting that is limiting
>>>>>>>> the size
>>>>>>>> > of memory allocated to the rules.
>>>>>>>> >
>>>>>>>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>>>>>>>> >
>>>>>>>> > Any suggestions are welcome.
>>>>>>>> >
>>>>>>>> > Gene
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Gene Albin
>>>>>>>> > gene.albin at gmail.com
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > _______________________________________________
>>>>>>>> > Oisf-users mailing list
>>>>>>>> > Oisf-users at openinfosecfoundation.org
>>>>>>>> >
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>> >
>>>>>>>> >
>>>>>>>> _______________________________________________
>>>>>>>> Oisf-users mailing list
>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> "Of course, someone who knows more about this will correct me if I'm
>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gene Albin
>>>>>> gene.albin at gmail.com
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> "Of course, someone who knows more about this will correct me if I'm
>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Oisf-users mailing listOisf-users at openinfosecfoundation.orghttp://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>
>>>>> In that respect.... What is your output of
>>>>> ulimit -aH
>>>>> and
>>>>> ulimit -a
>>>>> for the user that you run Suricata with?
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Peter Manev
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Gene Albin
>>>> gene.albin at gmail.com
>>>>
>>>>
>>>
>>>
>>> Hi Gene,
>>> I am using Suricata version 1.1beta2 (rev df3ca32).
>>> What happens if you try to run Suricata as root?
>>>
>>> Thanks
>>>
>>>
>>> --
>>> Peter Manev
>>>
>>
>>
>>
>> --
>> Gene Albin
>> gene.albin at gmail.com
>>
>>
>
>
> Sure,
> I did send mine to you.
> I tried with yours and Suricata is running aswell (I only modified the
> rules directory)
>
> Thanks
>
> --
> Peter Manev
>
Hi Gene
I think Dave is right - you might be running 2Gb/2Gb memory split on a
32bit.
I am running 1G / 3G memory split which gives me the edge I believe.
I am inclined to believe(please correct me if I am wrong ) that you have a
2Gb per process limit because of that (since you said yourself that Suricata
always gets killed when it reaches 2Gb of use at start up ) the PAE is not
going to help you in this case - if you would like to check that for sure
you could try 64bit system.
Thanks
--
Peter Manev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110731/ce088ef0/attachment-0002.html>
More information about the Oisf-users
mailing list