[Oisf-devel] Some questions about module development

[Victor Julien]

On 11/27/2012 04:50 PM, Jörg Vehlow wrote:
> Hi,
>  
> I am currently trying to develop some detection modules in order to
> detect non simple signature based malware cnc traffic for my master's
> thesis.
> The first module I implemented can calculate the entropy of the payload.
> And while doing that some questions (and possible answers) came up:
> 
> 1. What is the intended purpose of Setup, Match and Free functions of a
> detection module?
> I think Setup is for parsing the parameters, match does the actual
> calculating and matching and free is for freeing the memory allocated by
> Setup.
> 2. When are Setup, Match and Free called?
> From what I saw, I suspect that: Setup gets called ONCE for EVERY rule
> that uses the registered keyword at initialization and Free analog in
> the end. Match is called on every rule? Or does suricata match the
> keywords in order and if the first one fails it stops matching the other
> ones?

Setup is called per keyword occurrence. So if you have 2 rules with each
one "foo:bar;", Setup will be called twice.

Match is called on a per packet basis, but only if signature is actually
inspected (so not prefiltered based on flow dir, addresses, mpm and a
bunch of other conditions) and if the other conditions that are checked
prior to your keyword matched.

Free is called when we clean up our detection engine ctx. Normally at
shut down, but in case of rule parsing errors we can do it at init as
well. With live reload support it's also done at runtime, when the old
detect engine ctx is cleaned up.

> 
> 3. Where am I supposed to store my calculated values for reuse? I.e. if
> the entropy is used in more than one rule it would be cheaper to store
> the value somewhere.
> Do I have to add a member to the Packet structure?

That is an option. If you need it only for a single packet during one
inspection round you could also just use a thread local var, ie __thread
int yourint; or add something to the DetectEngineThreadCtx.

> 
> 4. What exactly is it I'm getting in Packet.payload?
> As far as I've seen it is the payload of the tcp / udp packet. Is it
> always the payload for the Application layer (app layer header and app
> layer payload)?

A packet payload, so the layer above the tcp/udp/sctp/icmp.

> 5. Is there a way to decrypt a packet on the fly but only if certain
> criteria are is matched, e.g. entropy greater x (I'd like to be able to
> do other matches like content on the decrypted code afterwards)?
> Can I just change Packet.payload to decrypt the packet and then go on
> matching on it? Or do I have to implement another app layer?

This is more tricky. Our content inspection engine is currently not
trivial to extend, it's something we will change. It's not very hard
either, but it requires a lot of small additions in a lot of places.

If you need the normal "content" inspection to work on your decrypted
payloads you will need to do your decryption before the detection engine
is invoked, so at the decode or applayer stages (or you can add your own
stage and hook it in before detect). You will have to store your
decrypted buffer in the packet or the flow, depending on the protocol,
then adapt the detect content inspection engine to know about it. Also,
for this you'd have to add a content modifier similar to http_uri,
http_client_body, etc.

> 6. This is about app layers: Are they always "executed"? What I mean is:
> Are all packets checked if they are possible http or smb even if there
> is no rules that needs this kind of matching?

Currently yes. We do some things on demand in the http parsers, but
otherwise yes.

> What I would like here is to have some match like entropy greater x and
> then execute an app layer decoder.

I think the best way currently would be to bail out early in the app
layer module if the entropy is below your threshold.

> 7. One last  (not so important) question: Why doesn't suricata use a
> real plugin system with dynamically linked plugins or something?
> Is it for performance reasons?

No, it actually on our (mental) roadmap. No ETA or anything yet :)

Cheers,
Victor

Btw, looks like you're not a member of this list, please subscribe.
-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------