[Oisf-users] Suricata and AI

[Cooper F. Nelson]

I wrote an expert system to automate escalation of suricata alerts to
our SOC.  It's based on a combination of keywords (the EmergingThreats
guys are pretty good at providing a standard taxonomy) and augmented
with some limited automated behavioral analysis for identifying new
threats.  For example, new SIDS, new domains serving packed
executables,  and patterns of multiple alerts within a sliding time window.

There are also multiple AV vendors that are using ML techniques and are
integrated with the VirusTotal service.  I believe that the
EmergingThreats project uses that as a malware source, so one could make
the case that AI is already producing threat intelligence that can be
leveraged by the suricata IDS engine.

There is also the generic problem that applying an ML approach to
'enumerating badness' doesn't always work out as anticipated.  For
example, I tried doing this years ago using simple static analysis
against packed executables; however to a ML classifier, compressed or
encrypted data is nothing more than white noise. 

An important thing to consider in this discussion is to collect some
metrics re: how effective the current EmergingThreats feed is at
detecting threats.  Of course, it's impossible to get the true answer to
this, but I did produce a report over the summer to get a general idea
of how well the current processes are working.  The executive summary is
that out of ~30k unique Emerging Threats PRO sigs, we are seeing about
10k unique alerts by SID per 30 day sample window.  So, in other words,
we are already looking for more threats than we are finding by a fair
margin, on a pretty big network with lots of BYOD systems and few
perimeter controls in the traditional sense. 

I have been considering taking an orthogonal approach and creating a
novel IDS engine implemented in golang that is designed from the ground
up to do ML-based detection.  But as mentioned, there are some hard
limits as to what can be accomplished and I'm not sure how well it will
work in general in this context.  There is also the issue that I don't
have easy access to an archive of malware packet captures to train it
against. 

tl;dr: The current suricata/ETPRO process already works pretty, pretty,
pretty, good. 

-Coop

On 11/2/2017 8:00 AM, Ale Fredes Hadad wrote:
> Hello eveyone!
>
> I am studying Suricata´s user guide and I didn´t find about Artificial
> Intelligence. Is there a plan to include any technique of artificial
> intelligence in Suricata? At present, do people combine Suricata with
> other tool to add AI?
>
> Anyone know if exists an article of a congress or something similar
> where they apply or combine techniques of Artificial Intelligence
> in/with Suricata?
> Thanks!
>
> Regrets,
> Alexis Fredes
>
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>
> Conference: https://suricon.net
> Trainings: https://suricata-ids.org/training/


-- 
Cooper Nelson
Network Security Analyst
UCSD ITS Security Team
cnelson at ucsd.edu x41042

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20171102/5d1d0cd6/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20171102/5d1d0cd6/attachment-0002.sig>