[Oisf-users] the challenge of IDS rules and your own db of maliciousness

[Victor Julien]

On 07/30/2014 12:55 PM, Christophe Vandeplas wrote:
> Thanks to MISP (notice the hidden advertisement) and the information
> sharing (about APTs) that happens more and more we are now slowly
> sitting on a bigger and bigger repository of IOCs related to APTs.
> Many of these are shared in private communities, but also many come
> from OSINT reports published by various companies. (APT1 as an
> example).
> 
> Sitting on data is not enough, so that's why MISP generates exports in
> various formats. From text, to csv, but also to Suricata rulesets.
> (also snort)
> 
> Now the problem I'm having is that this generates just to many rules
> for suricata to handle. As an example: Yesterday morning I had a rule
> file that contained 200 000 NIDS rules. This takes a huge amount of
> time to load into suricata and is not very efficient.
> 
> This huge amount of rules is cause by the variety of where the data
> can be used. As an example a hostname generates 3 rules: http , dns
> tcp, dns udp. Thanks to the Suricata protocol keywords this is reduced
> to 2 rules: http, dns. (this reduced the tules to 140 000 rules). But
> still it's too much for a NIDS.
> 
> As for the content of these rules, a quick grep | sed | sort | count
> magic gives me these counts:
>   39 498 Domain
>   88 198 Hostname
>   11 573 IP
>    2 816 URL
>    (less) ... other
> 
> There are multiple ways to reduce the number of rules loaded by the NIDS:
> - expiration of IOCs: easy in theory, difficult in practice, but we're
> working on this
> 
> - splitting detection over multiple "IDS" sources:  LogIDS (email,
> proxy, dns) and what the Log/SIEM does not see, load in the IDS.
> (we're doing this, but very inefficiently) . But then again, you miss
> things that did not use your proxy/relay server.

Splitting traffic may defeat Suricata's protocol detection, as you're
probably splitting on port (and ip's probably).

> - applying different concepts within the IDS: like the IP
> reputation/md5list that let's you load a file containing IOCs. However
> importing hostnames and domainnames in

Yeah, I would like to support this. In your case I think you have almost
a 100% exact matches. For this hash lookups would be fine. The advanced
rule logic isn't necessary. Then a single rule can be used, and using
the json output we could add _what_ we matched on. Not supported
currently, but I think this is the way forward.

> - bragging everywhere that you have a very valuable database, but not
> using it in detection, which is kinda sad  ;-)

Indeed.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------