[Oisf-users] HTTP Sessions and resource estimation

[Cooper F. Nelson]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm trying to figure out what the memory pig is here.

Could you drop the filter below in a file called 'http.bpf' and run suri
with this flag: '-F http.bpf'

> (not tcp src port 80 or (tcp src port 80 and ((tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 or tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x48545450))))

This will sample port 80 flows and should knock down your memory usage
if suri is operating correctly.  Otherwise you have some other problem.

- -Coop

On 3/30/2015 7:19 AM, Yasha Zislin wrote:
> I've changed the depth to 2mb and it didnt help. My memory utilization
> is the same.
> 
> 
> 
>> Date: Thu, 19 Mar 2015 11:22:16 -0700
>> From: cnelson at ucsd.edu
>> To: coolyasha at hotmail.com; oisf-users at lists.openinfosecfoundation.org
>> Subject: Re: [Oisf-users] HTTP Sessions and resource estimation
>>
> Say you have lots of customers watching streaming video over HTTP (like
> Netflix). Each video stream will ultimately consume 20MB of memory
> before suricata stops tracking it and releases the memory.
> 
> I also do a bit of performance analysis for HTTP proxy/cache design and
> it turns out that the vast majority (over 99%) of HTTP objects are under
> 1 MB in size, so you really aren't getting much from tracking past that.
> I understand that TCP connections are often left open and recycled, but
> most 'interesting' packets from a network security perspective are going
> to be within the first MB of new flows. In fact, most of the ET HTTP
> sigs (other than the WEB_CLIENT sigs) will only trigger against the
> first few packets, if at all.
> 
> -Coop
> 
> On 3/19/2015 10:54 AM, Yasha Zislin wrote:
>> Can you explain what it is? and how it affect memory utilization?
> 
>> Thanks.
> 
>>> Date: Thu, 19 Mar 2015 09:29:12 -0700
>>> From: cnelson at ucsd.edu
>>> To: coolyasha at hotmail.com; oisf-users at lists.openinfosecfoundation.org
>>> Subject: Re: [Oisf-users] HTTP Sessions and resource estimation
> 
>> I think that is too high a stream reassembly depth. Try 1mb instead.
> 
>> On 3/19/2015 7:19 AM, Yasha Zislin wrote:
> 
>>> My stream reassembly depth is set to 20mb. I forget why it is so high,
>>> but I've made it to minimize packet loss.
> 
>>> I am monitoring two span ports (about 1gig each) and my 40 logical
>>> CPUs/140 gigs of RAM server is using 95% of RAM.
>>> I thought Suricata was able to handle 10 gig feeds. Just trying to
>>> understand what I am doing wrong.
> 
>>> Thanks.
> 
> 
> 
> 

- -- 
Cooper Nelson
Network Security Analyst
UCSD ACT Security Team
cnelson at ucsd.edu x41042
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)

iQEcBAEBAgAGBQJVGXx4AAoJEKIFRYQsa8FWsd8H/jzkV0nQ8wdRDb4wFGiGhN6G
GtMW7jw3end/hTylqmp2B7r5i3Qhu0FrsiHxmvu0Fmleu0Wcx+tDpxModggB3F56
r1oKZmglsheAWH9bVbgheSweozKFMpdzv4vK4mfRmjqupNsyLmGME3wcUcRYvthI
qK1lLckwjs4qFkqpF02sTdUE05rHVKvILBEaxpMB2epPR6s20HxEDiTIdJhdxUKQ
c7SU2p2H3OdbzTZpY7mInA3PTFx9M7+gzXv5FM05v1CqHQjyGenhoaSMCZWP70XE
1JZDHjRsPpqI4L4tAfh8f7uX0X3wfi1qsQJlOHg7qcF35n7hM9+3l89+YC6jqkQ=
=x2Ij
-----END PGP SIGNATURE-----