[Oisf-devel] Storing suricata rules in database

[Anoop Saldanha]

On Mon, Oct 19, 2015 at 4:57 AM, Jason Ish <lists at unx.ca> wrote:
> On Sun, Oct 18, 2015 at 12:33 PM, ravin goyal <ravirocks1021 at gmail.com> wrote:
>>
>> On 18-Oct-2015 10:07 PM, "Jason Ish" <lists at unx.ca> wrote:
>>>
>>> On Fri, Oct 16, 2015 at 6:33 AM, ravin goyal <ravirocks1021 at gmail.com>
>>> wrote:
>>> > Hii all, I am working on suricata-2.0.8 and implemented it in IPS mode
>>> > to inspect ssl certificate , I am specifically want to inspect ssl
>>> > traffic and based upon the   tls subject field values, we are dropping
>>> > the packets.
>>> > Works pretty well
>>> >
>>> >
>>> > But I want to link database with suricata to store rules rather than
>>> > flat file structure.
>>> > I am going through the source code but I don't know where should I
>>> > begin my journey.
>>> >
>>> > I would appreciate if you provide an alternate solution to my
>>> > scenario, if my idea seems pretty broken.
>>>
>>> I'd try a simpler approach like a small tool that pulled the rules out
>>> of the database, wrote out the files and then sent Suricata a reload
>>> signal. If using PostgreSQL, have it run in the background, wait for
>>> Postgres notifications, write out the rules files and reload.
>>>
>>> I think I'd explore something like that before modifying Suricata.
>>
>> Thanks jason , I would try out it first as you have suggest, but my concern
>> is related to real time performance issues with flat file structure.
>> I want to eliminate the concept of file in it
>> As database keep on updating (with each read write operation) ,suricata
>> behaves accordingly( dropping packets as per the rules).
>> Wouldn't it be much simpler??
>
> Depends what you are trying to achieve.  I could be wrong, but it
> sounds like you are trying to make it more efficient to add and modify
> rules on the fly?  If so, I don't think (and someone please correct me
> if I am wrong), that Suricata can efficiently add one rule, or a rule
> modification.  Instead, to reload, even in the case where a single
> rule is modified, all rules have to be reloaded and then run through
> post-processing for proper setup and optimization.
>
> A quick check of loading the default ET Open ruleset on my machine
> shows about 25 seconds to read and load all the rules.  Of that, only
> a fraction of a second was spent reading from disk, the rest was all
> in rule processing which you aren't going to save on by moving to a
> database (if you'd even save in IO time).
>
> The latest code handles reloads much better then 2.0.9, so that might
> be worth a try.

Ravin,

What Jason said is right.  There is no single rule addition currently.
You still need to do a full rule reload.  Like Jason suggested you can
wait to see how fast the new latest rule reloads work.  How big is
your ruleset btw?  If you have a big ruleset it will good to share
your rule load nos too, with a profile of kind of rules you have.  The
data helps others who are trying something similar.

If you do feel that even that latest changes don't cut it wrt. rule
reload time, you can probably start looking at modifying the rule
reload staging code(have a look at SigGroupBuild) with the goal of
adding 1/x rules without a full reload.  You can then use suricata's
unix manager socket interface to load only specific rules.  The files
are definitely not an issue.

-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------