[Oisf-users] the challenge of IDS rules and your own db of maliciousness

[Christophe Vandeplas]

Hi there,


Thanks to MISP (notice the hidden advertisement) and the information
sharing (about APTs) that happens more and more we are now slowly
sitting on a bigger and bigger repository of IOCs related to APTs.
Many of these are shared in private communities, but also many come
from OSINT reports published by various companies. (APT1 as an
example).

Sitting on data is not enough, so that's why MISP generates exports in
various formats. From text, to csv, but also to Suricata rulesets.
(also snort)

Now the problem I'm having is that this generates just to many rules
for suricata to handle. As an example: Yesterday morning I had a rule
file that contained 200 000 NIDS rules. This takes a huge amount of
time to load into suricata and is not very efficient.

This huge amount of rules is cause by the variety of where the data
can be used. As an example a hostname generates 3 rules: http , dns
tcp, dns udp. Thanks to the Suricata protocol keywords this is reduced
to 2 rules: http, dns. (this reduced the tules to 140 000 rules). But
still it's too much for a NIDS.

As for the content of these rules, a quick grep | sed | sort | count
magic gives me these counts:
  39 498 Domain
  88 198 Hostname
  11 573 IP
   2 816 URL
   (less) ... other

There are multiple ways to reduce the number of rules loaded by the NIDS:
- expiration of IOCs: easy in theory, difficult in practice, but we're
working on this

- splitting detection over multiple "IDS" sources:  LogIDS (email,
proxy, dns) and what the Log/SIEM does not see, load in the IDS.
(we're doing this, but very inefficiently) . But then again, you miss
things that did not use your proxy/relay server.

- applying different concepts within the IDS: like the IP
reputation/md5list that let's you load a file containing IOCs. However
importing hostnames and domainnames in

- bragging everywhere that you have a very valuable database, but not
using it in detection, which is kinda sad  ;-)

- ...


I'm wondering what your experience is with using IOCs from sharing
with your IDS systems. Do you also have the problem of to many IDS
rules generated?
How do you solve it in your organisation?


Thanks a lot
Kind regards

Christophe