[Oisf-users] Memory Allocations

Wed Nov 26 14:19:52 UTC 2014

I would agree with you that my server is big for my traffic but nevertheless I keep getting closer to my RAM limit.I mean, PF RING takes a lot of RAM, my stream reassembly is huge and others.
So far it didnt crash for two days. I will keep on tweaking.
Thanks for all of your input.
I am posting back to community.

> Date: Tue, 25 Nov 2014 21:11:04 +0100
> Subject: Re: [Oisf-users] Memory Allocations
> From: petermanev at gmail.com
> To: coolyasha at hotmail.com
> 
> On Tue, Nov 25, 2014 at 7:34 PM, Yasha Zislin <coolyasha at hotmail.com> wrote:
> > Good to know about gaps. I guess if Suricata is not reporting packet loss
> > but gaps increase, it is for other reasons outside of our control.
> >
> > I use workers runmode. Latest PFRING. I've tested other modes and packet
> > capture and these two gave me the best results.
> > My packet loss is almost 0%. Occasionally, maybe once a day I get a burst of
> > packet loss of couple of million packets at once. Ideally I want it to be at
> > 0% but we get 10 mil packets per minute on both of the SPAN ports. If
> > Suricata can stay up without running out of RAM, these packet loss would be
> > probably acceptable.
> > I have two 10gig fiber nics : Broadcom Corporation Netxtreme II BCM57810
> >
> > I dont think I can use CPU affinity since it only applies to Intel cards. I
> > saw couple of your articles about that. I've tried it on mine, but such
> > settings.
> >
> > I have 2 x Intel Xeon CPU E5-2690 (3ghz) CPUs which makes 40 logical.
> > CPU doesnt get overutilized. I have 20 pfring threads per SPAN port. After
> > trying various combinations, this one was the best fit.
> 
> 
> This set up can handle 4x times the traffic in my opinion :) - so you
> should not be worried in that respect I think :)
> Please let the list know if you have overcome your OOM troubles or if
> you need any further help :)
> 
> Thank you
> 
> >
> >> Date: Mon, 24 Nov 2014 23:03:41 +0100
> >
> >> Subject: Re: [Oisf-users] Memory Allocations
> >> From: petermanev at gmail.com
> >> To: coolyasha at hotmail.com
> >>
> >> On Mon, Nov 24, 2014 at 9:19 PM, Yasha Zislin <coolyasha at hotmail.com>
> >> wrote:
> >> > Good to know. I will decrease my flow values and stream.
> >> > My goal is to avoid packet loss.
> >> > What kind of counters should I be looking at when stream memcap gets
> >> > maxed
> >> > out?
> >> > I get a lot of these:
> >> > tcp.reassembly_gap | RxPFReth219 | 473308
> >> >
> >>
> >> gaps are streams that have... gaps :) thus not allowing for proper
> >> inspection.
> >> Gaps can occur for various of reasons related to port mirroring and or
> >> traffic loss on the way to the mirror port and other...
> >>
> >> > As far as I know, these are lost packets.
> >> >
> >> > As far as my traffic goes:
> >> > - I monitor 2 SPAN ports (each is a 10gig card but traffic doesnt go
> >> > over
> >> > 2gigs).
> >> > - I use 20 logical CPUs for each SPAN Port.
> >> > - 97% of the traffic is HTTP
> >> >
> >>
> >> What do you use for runmode? afpacket/pfring?
> >> What is your packet loss? (percentage wise)
> >> What is your NIC model?(do you use CPU affinity)
> >> What are your CPUs?
> >>
> >> > So in my mind, stream, flow and timeouts are what is important and can
> >> > cause
> >> > packet loss.
> >> >
> >> > As far as RAM usage, suricata does crash. I get a lot of kernel messages
> >> > in
> >> > /var/log/messages when it dies.
> >> > Such as:
> >> > kernel: Out of memory: Kill process 4094 (RxPFReth04) score 982 or
> >> > sacrifice
> >> > child
> >> > Judging from TOP, SWAP barely gets used.
> >>
> >>
> >> yep ..that looks like OOM
> >>
> >>
> >> >
> >> > Oh, and I am not using 2.1 version of Suricata. I am on 2.0.4
> >> >
> >> > I appreciate your time helping me. Not everything is available in the
> >> > documentation. Trying to learn this wonderful IDS solution :)
> >> >
> >> >
> >> >
> >> >
> >> >> Date: Mon, 24 Nov 2014 20:00:03 +0100
> >> >
> >> >> Subject: Re: [Oisf-users] Memory Allocations
> >> >> From: petermanev at gmail.com
> >> >> To: coolyasha at hotmail.com
> >> >>
> >> >> On Mon, Nov 24, 2014 at 7:47 PM, Yasha Zislin <coolyasha at hotmail.com>
> >> >> wrote:
> >> >> > For reassembly, chunk-prealloc needs to be greater or equal than the
> >> >> > sum
> >> >> > of
> >> >> > all segements prealloc.
> >> >> > To get total reassembly ram needs sum up each segment's (size x
> >> >> > prealloc).
> >> >> > I've selected appropriate prealloc for each segment size after
> >> >> > running
> >> >> > suricata for a while and in debug mode it tells you if specific
> >> >> > segment
> >> >> > size
> >> >> > was exceeded.
> >> >> > So with these values, I need a bit under 40gb, which is fine.
> >> >> > For regular stream memcap, I have 2mil prealloc-sessions and i didnt
> >> >> > set
> >> >> > hash-size. I assume it is using default.
> >> >> > I keep decreasing my memcap and right now it is 10gb. Sounds like i
> >> >> > can
> >> >> > go
> >> >> > down all the way to like 3 gb.
> >> >>
> >> >> You should further lower it - more info why -
> >> >>
> >> >>
> >> >> http://pevma.blogspot.se/2014/08/suricata-flows-flow-managers-and-effect.html
> >> >>
> >> >>
> >> >> >
> >> >> > Now, I wonder because of which buffer my suricata runs out of memory.
> >> >>
> >> >> Suricata does not run out of memory. Suricata I suspect uses all the
> >> >> memory allocated to it ...which is more than what the box has - hence
> >> >> goes into swap.
> >> >>
> >> >> >
> >> >> > So after Suricata gets started and allocates whatever RAM for itself,
> >> >> > which
> >> >> > buffers are not fully allocated? It sounds like stream reassembly
> >> >> > buffer
> >> >> > doesnt get utilized right away.
> >> >> >
> >> >> > What about flows? isn't that kind of important? But judging from
> >> >> > stats.log I
> >> >> > dont have any negative entries for flow:
> >> >> > flow_mgr.closed_pruned | FlowManagerThread | 122808768
> >> >> > flow_mgr.new_pruned | FlowManagerThread | 23052773
> >> >> > flow_mgr.est_pruned | FlowManagerThread | 17538938
> >> >> > flow.memuse | FlowManagerThread | 11786200760
> >> >> > flow.spare | FlowManagerThread | 40006294
> >> >> > flow.emerg_mode_entered | FlowManagerThread | 0
> >> >> > flow.emerg_mode_over | FlowManagerThread | 0
> >> >>
> >> >> As long as you do not enter emergency mode you are ok.(so keep lowering
> >> >> it)
> >> >>
> >> >> >
> >> >> >
> >> >> > For config, I have
> >> >> > flow:
> >> >> > memcap: 15gb
> >> >> > hash-size: 3000000
> >> >> > prealloc: 40000000
> >> >> > emergency-recovery: 30
> >> >> >
> >> >>
> >> >>
> >> >> this is a seriously excessive setting for flow - please read my
> >> >> comments
> >> >> above.
> >> >>
> >> >> > And also, timeouts
> >> >> > flow-timeouts:
> >> >> >
> >> >> > default:
> >> >> > new: 3
> >> >> > established: 30
> >> >> > closed: 0
> >> >> > emergency-new: 10
> >> >> > emergency-established: 10
> >> >> > emergency-closed: 0
> >> >> > tcp:
> >> >> > new: 6
> >> >> > established: 100
> >> >> > closed: 0
> >> >> > emergency-new: 1
> >> >> > emergency-established: 5
> >> >> > emergency-closed: 2
> >> >> > udp:
> >> >> > new: 3
> >> >> > established: 30
> >> >> > emergency-new: 3
> >> >> > emergency-established: 10
> >> >> > icmp:
> >> >> > new: 3
> >> >> > established: 30
> >> >> > emergency-new: 1
> >> >> > emergency-established: 10
> >> >> >
> >> >>
> >> >> The timeouts are fine in my opinion. But then again I do not know what
> >> >> type of traffic you are monitoring and how much of it.
> >> >>
> >> >> > Thanks.
> >> >> >
> >> >> >
> >> >> >> Date: Mon, 24 Nov 2014 19:01:35 +0100
> >> >> >
> >> >> >> Subject: Re: [Oisf-users] Memory Allocations
> >> >> >> From: petermanev at gmail.com
> >> >> >> To: coolyasha at hotmail.com
> >> >> >>
> >> >> >> On Fri, Nov 21, 2014 at 4:25 PM, Yasha Zislin
> >> >> >> <coolyasha at hotmail.com>
> >> >> >> wrote:
> >> >> >> > Cool. Good to know that these are separate from each other.
> >> >> >> > So I understand how to calculate reassembly memcap (from your
> >> >> >> > article
> >> >> >> > I
> >> >> >> > believe).
> >> >> >> > I am not 100% clear how to calculate stream memcap.
> >> >> >>
> >> >> >> How did you calculate reassembly memcap?
> >> >> >>
> >> >> >>
> >> >> >> >
> >> >> >> > Also, I cant seem to figure out how much flow memcap I need. I
> >> >> >> > know
> >> >> >> > how
> >> >> >> > it
> >> >> >> > is being calculated with hash size and prealloc, but I dont know
> >> >> >> > how
> >> >> >> > much
> >> >> >> > hash-size and prealloc I need. Looking at stats.log, I always have
> >> >> >> > flow
> >> >> >> > spare values.
> >> >> >>
> >> >> >> Usually 1GB with 1mil prealloc is ok . Keep an eye on your emergency
> >> >> >> mode in your stats.log - as long as you do not enter that you are
> >> >> >> good.
> >> >> >> hash-size: 1048576
> >> >> >> prealloc: 1048576
> >> >> >>
> >> >> >> >
> >> >> >> > Can you advise me on these two?
> >> >> >> >
> >> >> >> > Thank you very much.
> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >> >> Date: Fri, 21 Nov 2014 09:44:04 +0100
> >> >> >> >
> >> >> >> >> Subject: Re: [Oisf-users] Memory Allocations
> >> >> >> >> From: petermanev at gmail.com
> >> >> >> >> To: coolyasha at hotmail.com
> >> >> >> >>
> >> >> >> >> On Thu, Nov 20, 2014 at 7:45 PM, Yasha Zislin
> >> >> >> >> <coolyasha at hotmail.com>
> >> >> >> >> wrote:
> >> >> >> >> > That's what I was asking in the original question :) do these
> >> >> >> >> > values
> >> >> >> >> > sum
> >> >> >> >> > up
> >> >> >> >> > or are part of each other?
> >> >> >> >> > so which one is used for segments? or what is the purpose of
> >> >> >> >> > both?
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> OK - apologies I didn't understand your question.
> >> >> >> >> Yes they are separate independent values.
> >> >> >> >>
> >> >> >> >> Reassembly memcap is with respect to all segments that need
> >> >> >> >> reassembly.
> >> >> >> >> Stream memcap is for .. stream :) - ones that are of no need for
> >> >> >> >> reassembly
> >> >> >> >>
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > On 20 nov 2014, at 19:34, Yasha Zislin <coolyasha at hotmail.com>
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> > Here is my section for stream
> >> >> >> >> >
> >> >> >> >> > stream:
> >> >> >> >> > memcap: 60gb
> >> >> >> >> > checksum-validation: no # reject wrong csums
> >> >> >> >> > inline: no # auto will use inline mode in IPS mode, yes
> >> >> >> >> > or no set it statically
> >> >> >> >> > prealloc-sessions: 2000000
> >> >> >> >> > midstream: false
> >> >> >> >> > asyn-oneside: false
> >> >> >> >> > reassembly:
> >> >> >> >> > memcap: 80gb
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > You have 60gb for stream and 80gb for reassembly = 140 ... You
> >> >> >> >> > are
> >> >> >> >> > right
> >> >> >> >> > away going into swap with this (if the memcaps are reached) as
> >> >> >> >> > compared
> >> >> >> >> > to
> >> >> >> >> > 132gb ram that you have on the box.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > depth: 20mb # reassemble 1mb into a stream
> >> >> >> >> > toserver-chunk-size: 2560
> >> >> >> >> > toclient-chunk-size: 2560
> >> >> >> >> > randomize-chunk-size: yes
> >> >> >> >> > #randomize-chunk-range: 10
> >> >> >> >> > #raw: yes
> >> >> >> >> > chunk-prealloc: 3000000
> >> >> >> >> > segments:
> >> >> >> >> > - size: 4
> >> >> >> >> > prealloc: 15000
> >> >> >> >> > - size: 16
> >> >> >> >> > prealloc: 200000
> >> >> >> >> > - size: 112
> >> >> >> >> > prealloc: 400000
> >> >> >> >> > - size: 248
> >> >> >> >> > prealloc: 300000
> >> >> >> >> > - size: 512
> >> >> >> >> > prealloc: 200000
> >> >> >> >> > - size: 768
> >> >> >> >> > prealloc: 100000
> >> >> >> >> > - size: 1448
> >> >> >> >> > prealloc: 1000000
> >> >> >> >> > - size: 65535
> >> >> >> >> > prealloc: 400000
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> > ________________________________
> >> >> >> >> > Subject: Re: [Oisf-users] Memory Allocations
> >> >> >> >> > From: petermanev at gmail.com
> >> >> >> >> > Date: Thu, 20 Nov 2014 19:32:51 +0100
> >> >> >> >> > To: coolyasha at hotmail.com
> >> >> >> >> >
> >> >> >> >> > Hi,
> >> >> >> >> >
> >> >> >> >> > What is the "reassembly" memcap?
> >> >> >> >> > Thanks
> >> >> >> >> >
> >> >> >> >> > Regards,
> >> >> >> >> > Peter Manev
> >> >> >> >> >
> >> >> >> >> > On 20 nov 2014, at 19:16, Yasha Zislin <coolyasha at hotmail.com>
> >> >> >> >> > wrote:
> >> >> >> >> >
> >> >> >> >> > I am running 2.0.4.
> >> >> >> >> >
> >> >> >> >> > Here are ALL memcap values:
> >> >> >> >> > defrag - 2gb
> >> >> >> >> > flow - 15gb
> >> >> >> >> > Stream - 60gb
> >> >> >> >> > Host - 10gb
> >> >> >> >> > === 87GB
> >> >> >> >> >
> >> >> >> >> > I've created such high memcap for Stream because it seems to
> >> >> >> >> > help
> >> >> >> >> > with
> >> >> >> >> > packet loss.
> >> >> >> >> > I am monitoring two interfaces with 10million packets per
> >> >> >> >> > minute
> >> >> >> >> > on
> >> >> >> >> > each
> >> >> >> >> > on
> >> >> >> >> > average. Most traffic is HTTP.
> >> >> >> >> > Can you recommend memcaps for each section?
> >> >> >> >> >
> >> >> >> >> > Thanks.
> >> >> >> >> >
> >> >> >> >> >
> >> >> >> >> >> Date: Thu, 20 Nov 2014 18:58:26 +0100
> >> >> >> >> >> Subject: Re: [Oisf-users] Memory Allocations
> >> >> >> >> >> From: petermanev at gmail.com
> >> >> >> >> >> To: coolyasha at hotmail.com
> >> >> >> >> >> CC: oisf-users at lists.openinfosecfoundation.org
> >> >> >> >> >>
> >> >> >> >> >> On Thu, Nov 20, 2014 at 6:50 PM, Yasha Zislin
> >> >> >> >> >> <coolyasha at hotmail.com>
> >> >> >> >> >> wrote:
> >> >> >> >> >> > I dont know if swap starts to be used by Suricata crashes
> >> >> >> >> >> > after
> >> >> >> >> >> > couple
> >> >> >> >> >> > of
> >> >> >> >> >> > days of running.
> >> >> >> >> >> > In system logs, I have kernel messages such as this:
> >> >> >> >> >> > kernel: RxPFReth22 invoked oom-killer: gfp_mask=0x201da,
> >> >> >> >> >> > order=0,
> >> >> >> >> >> > oom_adj=0,
> >> >> >> >> >> > oom_score_adj=0
> >> >> >> >> >> > kernel: RxPFReth22 cpuset=/ mems_allowed=0-1
> >> >> >> >> >> > kernel: Pid: 60417, comm: RxPFReth22 Not tainted
> >> >> >> >> >> > 2.6.32-504.el6.x86_64
> >> >> >> >> >> > #1
> >> >> >> >> >> >
> >> >> >> >> >> > Then after a ton of stack traces and memory errors, I see
> >> >> >> >> >> > this:
> >> >> >> >> >> > kernel: Out of memory: Kill process 59782 (Suricata-Main)
> >> >> >> >> >> > score
> >> >> >> >> >> > 985
> >> >> >> >> >> > or
> >> >> >> >> >> > sacrifice child
> >> >> >> >> >> > Killed process 59782, UID 501, (Suricata-Main)
> >> >> >> >> >> > total-vm:135646364kB,
> >> >> >> >> >> > anon-rss:108513440kB, file-rss:21329088kB
> >> >> >> >> >> >
> >> >> >> >> >> > I wouldnt be suprised that my buffers are set too big.
> >> >> >> >> >> > I am just not clear on some sections on how much RAM they
> >> >> >> >> >> > use.
> >> >> >> >> >> > and also for stream section, do you need to add memcap and
> >> >> >> >> >> > reassembly
> >> >> >> >> >> > buffers together or are they part of each other? As far as I
> >> >> >> >> >> > understand
> >> >> >> >> >> > reassembly buffer needs to be higher than memcap.
> >> >> >> >> >> >
> >> >> >> >> >> > I have 132gb of RAM. When suricata starts, it is using 64gb
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> Which Suricata version are you using?
> >> >> >> >> >> What is the total memcap sum values in your suricata.yaml?
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> >
> >> >> >> >> >> >> Date: Thu, 20 Nov 2014 18:21:54 +0100
> >> >> >> >> >> >> Subject: Re: [Oisf-users] Memory Allocations
> >> >> >> >> >> >> From: petermanev at gmail.com
> >> >> >> >> >> >> To: coolyasha at hotmail.com
> >> >> >> >> >> >> CC: oisf-users at lists.openinfosecfoundation.org
> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >> On Mon, Nov 17, 2014 at 3:45 PM, Yasha Zislin
> >> >> >> >> >> >> <coolyasha at hotmail.com>
> >> >> >> >> >> >> wrote:
> >> >> >> >> >> >> > I am having issues with Suricata crashing due to running
> >> >> >> >> >> >> > out
> >> >> >> >> >> >> > of
> >> >> >> >> >> >> > memory.
> >> >> >> >> >> >> > I just wanted to clarify certain sections of config that
> >> >> >> >> >> >> > I
> >> >> >> >> >> >> > am
> >> >> >> >> >> >> > doing
> >> >> >> >> >> >> > my
> >> >> >> >> >> >> > calculations correctly.
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > max-pending-packets 65000 ------- Does that use a lot of
> >> >> >> >> >> >> > Ram?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > So for defrag and flow sections, whatever memcap values I
> >> >> >> >> >> >> > set,
> >> >> >> >> >> >> > that's
> >> >> >> >> >> >> > what
> >> >> >> >> >> >> > the maximum that can be used, correct?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Stream section is a bit unclear to me. Memcap for Stream
> >> >> >> >> >> >> > and
> >> >> >> >> >> >> > Memcap
> >> >> >> >> >> >> > for
> >> >> >> >> >> >> > Reassembly, how do they relate? Which one should be
> >> >> >> >> >> >> > bigger?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Host section, once again, memcap is the maximum RAM that
> >> >> >> >> >> >> > would
> >> >> >> >> >> >> > be
> >> >> >> >> >> >> > used?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > And lastly, libhtp section, request and response
> >> >> >> >> >> >> > -body-limit
> >> >> >> >> >> >> > values,
> >> >> >> >> >> >> > is
> >> >> >> >> >> >> > that
> >> >> >> >> >> >> > maximum memory utilization of LIBHTP?
> >> >> >> >> >> >> >
> >> >> >> >> >> >> > Thanks.
> >> >> >> >> >> >> >
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> Hi,
> >> >> >> >> >> >>
> >> >> >> >> >> >> You mean you are running into swap, correct?
> >> >> >> >> >> >>
> >> >> >> >> >> >> If you sum up all the memcap values you have given in
> >> >> >> >> >> >> suricata.yaml
> >> >> >> >> >> >> -
> >> >> >> >> >> >> would that be less than what you actually have as RAM on
> >> >> >> >> >> >> the
> >> >> >> >> >> >> server
> >> >> >> >> >> >> running Suricata?
> >> >> >> >> >> >>
> >> >> >> >> >> >> Thank you
> >> >> >> >> >> >>
> >> >> >> >> >> >>
> >> >> >> >> >> >> --
> >> >> >> >> >> >> Regards,
> >> >> >> >> >> >> Peter Manev
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >>
> >> >> >> >> >> --
> >> >> >> >> >> Regards,
> >> >> >> >> >> Peter Manev
> >> >> >> >>
> >> >> >> >>
> >> >> >> >>
> >> >> >> >> --
> >> >> >> >> Regards,
> >> >> >> >> Peter Manev
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >> Peter Manev
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Peter Manev
> >>
> >>
> >>
> >> --
> >> Regards,
> >> Peter Manev
> 
> 
> 
> -- 
> Regards,
> Peter Manev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20141126/3af53556/attachment-0002.html>