[Oisf-users] Suricata runs out of memory on startup
Dave Remien
dave.remien at gmail.com
Sun Jul 31 15:36:49 UTC 2011
Peter,
Gene has a 1G/3G kernel and can indeed get 3G; I sent him a quickie prog
that shows that. After that, seems like valgrind is in order to try to
figure out where the extra memory is being allocated, if you can't see why
from the suri.yaml files.... possibly an area Victor could shed light on.
Or, as you say, move to a 64 bit kernel. At least there Gene can get 4G -
8-).
Cheers,
Dave
On Sun, Jul 31, 2011 at 5:09 AM, Peter Manev <petermanev at gmail.com> wrote:
>
>
> On Sat, Jul 30, 2011 at 5:49 PM, Peter Manev <petermanev at gmail.com> wrote:
>
>>
>>
>> On Fri, Jul 29, 2011 at 5:34 PM, Gene Albin <gene.albin at gmail.com> wrote:
>>
>>> Peter,
>>> Sudo suricata ends with the same problem. I'm wondering if there may
>>> be a difference between the optimization of our suricata.yaml files. Would
>>> you mind exchanging files so that we can compare?
>>>
>>> Gene
>>>
>>>
>>> On Fri, Jul 29, 2011 at 1:05 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>
>>>>
>>>>
>>>> On Thu, Jul 28, 2011 at 8:59 PM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>
>>>>> Gents,
>>>>> Thanks all for the suggestions. Specific responses below
>>>>>
>>>>> Rmkml - I made your recommended change to the suricata.yaml file (-
>>>>> profile: low) and there was no change. Watching top the spike appeard to
>>>>> peak at 3.7GB then Suricata exited.
>>>>>
>>>>> Dave - I have to admit that I'm not versed in C so I can't build the
>>>>> program you're describing. Do you know where I could find a pre-built one?
>>>>> I agree that upgrading to a 64-bit version would be the best choice, however
>>>>> I'm up against a very short timeline here and won't have time to upgrade the
>>>>> OS the reinstall Suricata and the rest of the tools that I need.
>>>>>
>>>>> Peter - Here are the results from both ulimit -aH and ulimit -a:
>>>>>
>>>>> [gene at suri2 ~]$ ulimit -aH
>>>>> core file size (blocks, -c) unlimited
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 278528
>>>>> max locked memory (kbytes, -l) 32
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 1024
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) unlimited
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 278528
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> [gene at suri2 ~]$ ulimit -a
>>>>> core file size (blocks, -c) 0
>>>>> data seg size (kbytes, -d) unlimited
>>>>> scheduling priority (-e) 0
>>>>> file size (blocks, -f) unlimited
>>>>> pending signals (-i) 278528
>>>>> max locked memory (kbytes, -l) 32
>>>>> max memory size (kbytes, -m) unlimited
>>>>> open files (-n) 1024
>>>>> pipe size (512 bytes, -p) 8
>>>>> POSIX message queues (bytes, -q) 819200
>>>>> real-time priority (-r) 0
>>>>> stack size (kbytes, -s) 10240
>>>>> cpu time (seconds, -t) unlimited
>>>>> max user processes (-u) 278528
>>>>> virtual memory (kbytes, -v) unlimited
>>>>> file locks (-x) unlimited
>>>>>
>>>>> I'm not exactly sure what I'm looking at here, but it looks like my max
>>>>> memory size is unlimited. Am I reading this correctly?
>>>>> I've also looked over your screen capture from your Ubuntu 32-bit VM
>>>>> and can't figure out why I'm crashing. What version of Suricata are you
>>>>> running on that 32-bit Ubuntu VM? 1.1b2?
>>>>>
>>>>> Gene
>>>>>
>>>>>
>>>>> On Thu, Jul 28, 2011 at 7:14 AM, Peter Manev <petermanev at gmail.com>wrote:
>>>>>
>>>>>> **
>>>>>> On 07/28/2011 02:54 PM, Dave Remien wrote:
>>>>>>
>>>>>> If you're up for it, about 15 lines of C code will give you a tiny
>>>>>> program to test how much memory you can get for a single process - basically
>>>>>> just malloc in a loop until you can't anymore. Sounds like your environment
>>>>>> may actually be limited to 2GB of process size; normal for Linux is 3GB (all
>>>>>> in the 32 bit world). Or you could lobby for a 64 bit copy
>>>>>> of Centos; that'll eliminate the cap (for this purpose).
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> Dave
>>>>>>
>>>>>> On Thu, Jul 28, 2011 at 1:10 AM, Gene Albin <gene.albin at gmail.com>wrote:
>>>>>>
>>>>>>> I just created a ticket with the details. To answer the questions
>>>>>>> here, I'm running the 1.1b2 build from the tarball. Not using git. The
>>>>>>> machine is running the 32 bit version of CentOS5.6, but we just applied the
>>>>>>> kernel-PAE packages today to allow it to utilize more than 4GB of ram. Is
>>>>>>> this what you are talking about, Dave? Lastly I included the suricata.yaml
>>>>>>> file as well as the output from free -m and my collectl memory statistics
>>>>>>> during the fatal run.
>>>>>>>
>>>>>>> Thanks for helping out with this. I thought that bumping the ram up
>>>>>>> to 16GB would fix it, but it appears not. Maybe I'll start slicing off some
>>>>>>> rules and see where the threshold lies...
>>>>>>>
>>>>>>> Gene
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 27, 2011 at 7:44 PM, Dave Remien <dave.remien at gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 27, 2011 at 5:02 PM, Will Metcalf <
>>>>>>>> william.metcalf at gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Can you create a redmine ticket and attach a scrubbed version of
>>>>>>>>> your
>>>>>>>>> suricata.yaml? Along with output of free -m prior to starting
>>>>>>>>> suri?
>>>>>>>>>
>>>>>>>>
>>>>>>>> Are you running a 32 bit kernel with a 2GB/2GB memory split, by
>>>>>>>> any chance??
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>>
>>>>>>>> Dave
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://redmine.openinfosecfoundation.org/projects/suricata
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>>
>>>>>>>>> Will
>>>>>>>>> On Wed, Jul 27, 2011 at 4:35 PM, Gene Albin <gene.albin at gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> > Ok, I'm probably doing something wrong here, but every time I
>>>>>>>>> try to load a
>>>>>>>>> > combined rule file with all of the VRT and ET rules enabled (~30K
>>>>>>>>> rules) it
>>>>>>>>> > fails following stage 3:
>>>>>>>>> >
>>>>>>>>> > [7069] 27/7/2011 -- 14:14:09 - (detect.c:631) <Info>
>>>>>>>>> (SigLoadSignatures) --
>>>>>>>>> > 102 rule files processed. 30183 rules succesfully loaded, 164
>>>>>>>>> rules failed
>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2161) <Info>
>>>>>>>>> > (SigAddressPrepareStage1) -- 30701 signatures processed. 1800 are
>>>>>>>>> IP-only
>>>>>>>>> > rules, 20152 are inspecting packet payload, 11088 inspect
>>>>>>>>> application layer,
>>>>>>>>> > 0 are decoder event only
>>>>>>>>> > [7069] 27/7/2011 -- 14:14:47 - (detect.c:2164) <Info>
>>>>>>>>> > (SigAddressPrepareStage1) -- building signature grouping
>>>>>>>>> structure, stage 1:
>>>>>>>>> > adding signatures to signature source addresses... complete
>>>>>>>>> > [7069] 27/7/2011 -- 14:14:48 - (detect.c:2806) <Info>
>>>>>>>>> > (SigAddressPrepareStage2) -- building signature grouping
>>>>>>>>> structure, stage 2:
>>>>>>>>> > building source address list... complete
>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3363) <Info>
>>>>>>>>> > (SigAddressPrepareStage3) -- MPM memory 1801173581 (dynamic
>>>>>>>>> 1801173581, ctxs
>>>>>>>>> > 0, avg per ctx 0)
>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3365) <Info>
>>>>>>>>> > (SigAddressPrepareStage3) -- max sig id 30701, array size 3838
>>>>>>>>> > [7069] 27/7/2011 -- 14:16:40 - (detect.c:3376) <Info>
>>>>>>>>> > (SigAddressPrepareStage3) -- building signature grouping
>>>>>>>>> structure, stage 3:
>>>>>>>>> > building destination address lists... complete
>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>> <Error>
>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] -
>>>>>>>>> SCMalloc
>>>>>>>>> > failed: Cannot allocate memory, while trying to allocate 558852
>>>>>>>>> bytes
>>>>>>>>> >
>>>>>>>>> > [7069] 27/7/2011 -- 14:16:43 - (detect-engine-siggroup.c:1583)
>>>>>>>>> <Error>
>>>>>>>>> > (SigGroupHeadBuildHeadArray) -- [ERRCODE: SC_ERR_FATAL(169)] -
>>>>>>>>> Out of
>>>>>>>>> > memory. The engine cannot be initialized. Exiting...
>>>>>>>>> >
>>>>>>>>> > I have done this while watching the memory useage in top (set to
>>>>>>>>> refresh
>>>>>>>>> > every .2 seconds). Initially when this happened I only had 4GB
>>>>>>>>> allocated to
>>>>>>>>> > the VM. Useage never gets beyond 2GB so that left almost 2GB
>>>>>>>>> available. I
>>>>>>>>> > decided to bump the VM up to 8GB but the problem didn't go away.
>>>>>>>>> It still
>>>>>>>>> > exits when the memory useage gets to around 2GB.
>>>>>>>>> >
>>>>>>>>> > Everything works fine when I load a reduced ruleset, i.e. just
>>>>>>>>> VRT or just
>>>>>>>>> > ET, but for my tests I want to load both. Before I go back to
>>>>>>>>> the VM
>>>>>>>>> > administrator and ask for 16 GB (and wait several days for the
>>>>>>>>> allocation) I
>>>>>>>>> > was wondering if there might be a config setting that is limiting
>>>>>>>>> the size
>>>>>>>>> > of memory allocated to the rules.
>>>>>>>>> >
>>>>>>>>> > Running 1.1b2 on CentOS 5.6 - 4core VMWare ESXi.
>>>>>>>>> >
>>>>>>>>> > Any suggestions are welcome.
>>>>>>>>> >
>>>>>>>>> > Gene
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Gene Albin
>>>>>>>>> > gene.albin at gmail.com
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > _______________________________________________
>>>>>>>>> > Oisf-users mailing list
>>>>>>>>> > Oisf-users at openinfosecfoundation.org
>>>>>>>>> >
>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> _______________________________________________
>>>>>>>>> Oisf-users mailing list
>>>>>>>>> Oisf-users at openinfosecfoundation.org
>>>>>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> "Of course, someone who knows more about this will correct me if I'm
>>>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Gene Albin
>>>>>>> gene.albin at gmail.com
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> "Of course, someone who knows more about this will correct me if I'm
>>>>>> wrong, and someone who knows less will correct me if I'm right."
>>>>>> David Palmer (palmer at tybalt.caltech.edu)
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Oisf-users mailing listOisf-users at openinfosecfoundation.orghttp://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>>>>>
>>>>>> In that respect.... What is your output of
>>>>>> ulimit -aH
>>>>>> and
>>>>>> ulimit -a
>>>>>> for the user that you run Suricata with?
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Peter Manev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gene Albin
>>>>> gene.albin at gmail.com
>>>>>
>>>>>
>>>>
>>>>
>>>> Hi Gene,
>>>> I am using Suricata version 1.1beta2 (rev df3ca32).
>>>> What happens if you try to run Suricata as root?
>>>>
>>>> Thanks
>>>>
>>>>
>>>> --
>>>> Peter Manev
>>>>
>>>
>>>
>>>
>>> --
>>> Gene Albin
>>> gene.albin at gmail.com
>>>
>>>
>>
>>
>> Sure,
>> I did send mine to you.
>> I tried with yours and Suricata is running aswell (I only modified the
>> rules directory)
>>
>> Thanks
>>
>> --
>> Peter Manev
>>
>
> Hi Gene
>
> I think Dave is right - you might be running 2Gb/2Gb memory split on a
> 32bit.
> I am running 1G / 3G memory split which gives me the edge I believe.
> I am inclined to believe(please correct me if I am wrong ) that you have a
> 2Gb per process limit because of that (since you said yourself that Suricata
> always gets killed when it reaches 2Gb of use at start up ) the PAE is not
> going to help you in this case - if you would like to check that for sure
> you could try 64bit system.
>
> Thanks
> --
> Peter Manev
>
--
"Of course, someone who knows more about this will correct me if I'm
wrong, and someone who knows less will correct me if I'm right."
David Palmer (palmer at tybalt.caltech.edu)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110731/ad9124ba/attachment-0002.html>
More information about the Oisf-users
mailing list