[Oisf-devel] tcp.ssn_memcap_drop

[Martin Holste]

Now I've reduced the ruleset to a single rule--my heartbeat sig, and
it's still missing almost all the time.  It appears that there's a
major bottleneck in the flow distributer somewhere if the system can't
grep for a single stream on just 600 Mb/sec.  Anyone else running
heartbeat sigs and seeing the same thing?

On Mon, Sep 19, 2011 at 11:06 AM, Martin Holste <mcholste at gmail.com> wrote:
> Ok, I'm giving that a shot, but so far that doesn't seem to have
> improved things.  Right now, it looks like the system is missing a ton
> of heartbeats, so it's definitely not detecting everything even though
> all the drop counters are zero.  I'm running just 3k signatures on
> about 600 Mb/sec of HTTP on 8 CPU/16 GB system.
>
> On Mon, Sep 19, 2011 at 10:48 AM, Victor Julien <victor at inliniac.net> wrote:
>> On 09/19/2011 05:43 PM, Martin Holste wrote:
>>> I've got memcap at 4GB and max_sessions is 256k by default.  I'm
>>
>> You may want to try setting it a bit lower than the 4GB max, like 3.5GB
>> or so. I think I've seen at least one occasion where it didn't behave
>> properly with the max setting. Something we need to look into still.
>>
>> Cheers,
>> Victor
>>
>>> having better luck now with more drastic emergency flow pruning:
>>>
>>> flow:
>>>   #memcap: 33554432
>>>   memcap: 4294967295
>>>   #hash_size: 65536
>>>   hash_size: 268435456
>>>   prealloc: 10000
>>>   emergency_recovery: 40 #30
>>>   prune_flows: 500 #5
>>>
>>> flow-timeouts:
>>>   default:
>>>     new: 1 # 30
>>>     established: 10 #300
>>>     closed: 0
>>>     emergency_new: 1 #10
>>>     emergency_established: 1 #100
>>>     emergency_closed: 0
>>>   tcp:
>>>     new: 1 #60
>>>     established: 10 #3600
>>>     closed: 120
>>>     emergency_new: 1 #10
>>>     emergency_established: 1 #300
>>>     emergency_closed: 20
>>>   udp:
>>>     new: 1 #30
>>>     established: 1 #300
>>>     emergency_new: 1 #10
>>>     emergency_established: 1 #100
>>>   icmp:
>>>     new: 1 #30
>>>     established: 1 #300
>>>     emergency_new: 1 #10
>>>     emergency_established: 1 #100
>>>
>>> I'm not yet sure how this will affect detection, but prior to this,
>>> most new flows were being discarded.  This policy should favor new
>>> flows at the expense of old flows, which for malware detection should
>>> be desired.
>>>
>>> On Mon, Sep 19, 2011 at 10:30 AM, Anoop Saldanha <poonaatsoc at gmail.com> wrote:
>>>> stream:
>>>>  memcap: 33554432              # 32mb
>>>>
>>>> At the same time, you might also want to set max_sessions to something
>>>> bigger.  We default to 256k.  You can try a bigger no and see how that
>>>> works out
>>>>
>>>> On Mon, Sep 19, 2011 at 8:07 PM, Martin Holste <mcholste at gmail.com> wrote:
>>>>> I'm seeing a ton of tcp.ssn_memcap_drop in my stats.log.  Which memcap
>>>>> do I need to tweak to decrease these drops?  I've already set them all
>>>>> to 4GB.
>>>>> _______________________________________________
>>>>> Oisf-devel mailing list
>>>>> Oisf-devel at openinfosecfoundation.org
>>>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Anoop Saldanha
>>>>
>>> _______________________________________________
>>> Oisf-devel mailing list
>>> Oisf-devel at openinfosecfoundation.org
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>>
>>
>>
>> --
>> ---------------------------------------------
>> Victor Julien
>> http://www.inliniac.net/
>> PGP: http://www.inliniac.net/victorjulien.asc
>> ---------------------------------------------
>>
>> _______________________________________________
>> Oisf-devel mailing list
>> Oisf-devel at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>>
>