[Oisf-users] EXTERNAL: Re: Making Suricata Alert Per Matching Packet

[Victor Julien]

Hi all,

On 12-04-16 18:27, Cooper F. Nelson wrote:
> I suspect it's just that the stream tracker is designed to only generate
> one alert per flow.  I would even say this is expected behavior.

As this is a common source of confusion, here's a little overview. TL;DR
it's complicated.


Suricata has different classes of rules

1. ip-only rules. Rules that only inspect header properties that stay
static over the lifetime of a (bidirectional) flow are inspected in a
separate inspection engine. This will only inspect the first packet in
each flow direction.

Rule keywords like ttl will exclude a rule from the ip-only engine, as
this is a per packet property. Also if the if the flow engine runs out
of memory, we fall back to per packet inspection.


2. stateful rules

a. tx aware

Transaction based protocols (e.g. HTTP/1.1) will inspect protocol aware
rules once per transaction. In the HTTP example a http_uri rule will be
inspected again each TX, so against each URI. If a rule has a URI match
and a BODY match, the state of this matching process is stored in the
flow structure so that the URI can match when we see packet X and the
BODY when we see packet X+1000.

b. other

A couple of protocol implementations are not tx-aware (e.g. ssh). These
will match max once per flow.


3. raw content inspection

Raw content inspection means the pattern matching and other content
inspection keywords (e.g. content, isdataat, pcre) that are not used to
look at a HTTP buffer or similar, but simply at packet and reassembled
stream payloads.

A simple rule like 'alert tcp any any -> any any (content:"abc";) will
by default be inspected against the reassembled stream only. However,
one exception is that when the stream engine wasn't able to add a packet
to it's state, the packet payload is inspected.

Some rule keywords will force the inspection against packets. For
example 'dsize' is only valid for packets, so if this appears in a rule
the rule will only be evaluated against the packet payload. Another is
'alert tcp-pkt ... '. The opposite is also possible, 'alert tcp-stream
...' will force only stream inspection.

The stream inspection happens in chunks that are larger than a single
packet generally. This is controlled by toserver-chunk-size and it's
toclient opposite in the stream config section.

For per packet payload inspection, those rules are inspected per packet
that has a payload.


To further complicate matters there is the mpm stage. The multi pattern
matching stage (some refer to it as fast_pattern) makes sure that only
those rules that have a remote chance of matching are evaluated
individually. So a rule may not appear to be inspected at all, but this
is because the fast pattern of the rule didn't appear in the payload.

To even further complicate things, Suricata does more 'prefilter' things
to avoid having to look at rules individually.


4. the rest

The rest should be rules that for example look at the ttl field. Those
should be inspected against all packet that have a remote chance of
matching. So a tcp specific keyword (flags) will not be inspected
against UDP packets.



Some helpful tools:

Runtime option --list-keywords=all will give you list including which
keywords are 'ip-only compatible'.

Runtime option --engine-analysis will dump some info about the loaded
rules into your log directory

Compile time option --enable-profiling will give you a lot more info
about how often rules are inspected. There are per rule, per rule
keyword and packet profiling outputs.


I'm open to ideas how we can make this easier to grasp (other than
documentation). I'd be happy to create some new analysis output to give
more insights into how Suri sees a rule.

Cheers,
Victor


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------