[Oisf-users] Question Regarding Prune Statistics

[Victor Julien]

On 04/09/2015 09:19 PM, Rasmor, Zachary R wrote:
> I am trying to make sense of the positive “pruned” values that I’m
> seeing in my stats log. My understanding is that positive prune values
> are undesirable; however my understanding was also that pruning occurs
> in emergency mode after shortened timeouts have not stabilized memory usage…
> 
>  
> 
> Per
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/Suricatayaml:
> 
>  
> 
> “At the point the memcap will still be reached, despite prealloc, the
> flow-engine goes into the emergency-mode. In this mode, the engine will
> make use of shorter time-outs. It lets flows expire in a more aggressive
> manner so there will be more space for new Flows. 
> There are two options: emergency_recovery and prune_flows. The emergency
> recovery is set on 30. This is the percentage of prealloc'd flows after
> which the flow-engine will be back to normal (when 30 percent of the
> 10000 flows is completed).
> If during the emergency-mode, the aggressive time-outs do not have the
> desired result, this option is the final resort. It ends some flows even
> if they have not reached their time-outs yet. The prune-flows option
> shows how many flows there will be terminated at each time a new flow is
> set up.”
> 
>  
> 
> I have pasted the final stats entry of a 30 min test below. As you can
> see, I never entered emergency mode, however my positive prune statistics.
> 
> (One additional note about the passage above: the “prune-flows” setting
> seems to have been deprecated as of v1.3.1)
> 
>  
> 
> flow_mgr.closed_pruned    | FlowManagerThread         | 5016163
> 
> flow_mgr.new_pruned       | FlowManagerThread         | 1121133
> 
> flow_mgr.est_pruned       | FlowManagerThread         | 1885848
> 
> flow.memuse               | FlowManagerThread         | 1433762256
> 
> flow.spare                | FlowManagerThread         | 2097144
> 
> flow.emerg_mode_entered   | FlowManagerThread         | 0
> 
> flow.emerg_mode_over      | FlowManagerThread         | 0
> 
>  
> 
> This leads to a few questions:
> 

These counters are incremented when the flow manager (which acts as an
async garbage collector for the flow hash) evicts a flow from the hash.
The 3 counters new_, est_, closed_ indicate the flow state at the time
of the eviction.

> 
> 1.       Are positive pruned values necessarily a bad thing?

No. They are completely normal and expected.

> 2.       Per the Suricata training class: “new_pruned means flow were
> discarded before they were established” – what is the meaning of
> est_pruned and closed_pruned?

est_ means a flow was evicted when in the 'established' state. This means:

1. in tcp, we are after the 3whs, but before shutdown (rst/fin)
2. in all other protocols: we've seen traffic from both sides on the
'flow' (so both from server and from client)

> 3.       How is pruning occurring if we never enter emergency mode?

The pruning happens when flows time out. Timing out here means that we
haven't seen a new packet for a flow in X seconds. The value of X
depends on 3 things: proto, flow state (new, est, closed) and emergency
mode. The values are taken from your yaml, the flow-timeouts section.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------