[Oisf-devel] Draft Response to Victor

[Victor Julien]

On 06/03/2015 09:56 AM, Anoop Saldanha wrote:
> Once this initial version is working, the next thing we'd like to do is see
> if we can insert the new rules into the existing detection engine without
> reparsing all the other rules or disturbing their current state. The goal
> would be to get the new rules (and only the new rules) parsed, get any
> associated state initialized, and then insert them into the rule list in the
> running detection engine, pausing it only long enough to update the list
> pointers. We do not want to disturb the state of the currently running rules
> and allow them to continue processing packets & flows uninterrupted (or as
> much so as possible).

This won't be possible w/o a whole bunch of rewriting/refactoring.

A DetectEngineCtx is supposed to be static after setting it up so that
multiple threads can use it w/o locking. There are loads of cross
references within the structure, pointers or array indexes. Also, the
per thread DetectEngineThreadCtx depends in a number of ways on the
DetectEngineCtx not changing.

The current reload code creates a new DetectEngineCtx plus it's per
thread DetectEngineThreadCtx instances and then per thread hotswaps the
DetectEngineThreadCtx pointer. Existing per flow state in stateful rules
is then reset, as it depends (partly) on pointers to the old ctx (this
is btw something that is not completely necessary anymore, maybe).

Just a few things that we do after the rules are loaded, but before the
detect engine is used:

- sorting: rules are sorted by priority and a few other things, like
setting/checking flowbits
- grouping: rules are grouped based on protocol, addresses, ports, etc
- fast_pattern/mpm: mpm ctx' are built based on patterns in the set or
per rule group
- ip-only: radix tree is built
- etc

Updating this atomically w/o interrupting the packet paths isn't
possible right now.

Still, in the longer run this is a goal for me as well. But it will
require a lot of work, that will likely have to be done in smaller
steps. E.g. first do 'independent' ip-only rules. Independent meaning no
dependencies on flowbits and such, so no sorting would be needed.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------