[Oisf-devel] RFC: extensible alert format

[Eric Leblond]

Hello,

Here's a proposal for an enhanced alert format. Fell free to comment.

============================
EAF: Extensible Alert Format
============================

Principle
=========

Current alert system does only provide poor information. For example, in
case of
stream matching we don't output properly the stream. Or in case of a
match giving
some meaningful information there is no output of what was matched.

To solve these problems, a solution is to define a new alert format that
can
contain an arbitrary number of information and be extensible.

A good candidate is to use JSON for that. We can output the basic
alert fields and adds information from the matching module. For example,
http_user_agent could ouput the user agent in case of a match or
a pcre match could return the string matched.

In the case of binary content, we can use base64 encoding to protect
the string.

An example alert could looks like ::

 { 
   "timestamp": "2013...", "srcip":"::1", "dstip":"::2", "proto":"tcp",
   "sid":"121213", "msg":"rhooo sexy",
   "stream": {
            "encoding":"base64",
            "content":"3294928492DUH29UE29294892849284"
                  },
   "pcre": { "match": "efjehfeuh"},
   "http_user_agent" { "value": "WGET"}
 }

Maybe sig can contains information about weither we want to log or not
the extended fields. Maybe we can add a content modifier like ''nolog''
when there is no interest on the value match in the field.

Luajit
------

Luajit should be able to export a key value dictionnary to be added into
the
alert message.


Implementation
==============
Algorithm
---------

Update Match and MAtchApp function in signature to update the
Packet structure if a match occur.

This requires an update of the packet structure to contain
matches. We can use a field named matches. The Match function
can then add information inside the field.

We will need inside of this field:

- sid: needed to output the matches during the alert (which got sid
info)
- A key - type - value storage able to store an arbitrary number of
values

It seems a good idea to use JSON for the key type value storage as this
will be the main output.

So we can have a linked list of structure containing a sid and a json_t
object.

If there is no match on sig we need to clean the matches structure for
sid that did not match. When outputing alert, we get the Matches 
corresponding to the sig ID and we delete and clean the element once
logging is done.

Details
-------

Getting a json object may be interesting even more complex thing can be
handled.
One other advantage is that a direct use of jansson will allow us to to
benefit
from allocation system.

In the case of implementation of queueing system for alerting, the
json_t object
can be simply passed as a pointer to the generated alert message.

Performances
============

This trigger some allocations that are unnecessary if there is no match
at the end
So it should only be used for final matching. 

BR,
-- 
Eric Leblond <eric at regit.org>
Blog: https://home.regit.org/