[Oisf-users] Re : Record traffic as soon as a thread is detected

[Kerry Milestone]

Think of ES as a document store, rather than a database and you'll move
away from thinking about new/old.  Discover the wonderful world of
sharding, Lucene, clusters, production pipelines...

It is extremely simple to take JSON and insert it into a fully indexed
and quick retrieval platform.  On the the way through, you could tag
addresses with a business group (from a lookup) and implement a query
interface so each business group (using auth from flavour of the day or
what ever the business already uses for SSO) which permits access to
their specific records which must contain this tag.

for instance, the rather verbose output you can enable:
http://Suricata.readthedocs.io/en/latest/output/eve/eve-json-output.html

It does work well using Suricata with redis, setup a listener in the
loop which starts n2disk when specific alerts or addresses are triggered
- and fire a message to ES to store in the metadata (ie flow_id) about
the capture alongside the Suricata alerts picked up from the same redis
loop.  If using PF_RING and Suricata there is very little overhead to
doing this.  You'll need to {cough} remember to stop (or automate or
kill after n time) these captures somehow... or else you'll fill the disk...

More than likely, you could do all this within Suricata with lua - I've
not tried.  Also, Suricata can send n bytes of a pcap with each alert
which for many things, such as DNS alerts, is adequate for a bit further
investigation.

It is reasonably trivial to setup basic ES and ingest all the Suricata
messages.  Depending on the scale of your network and processing
resources, Suricata can also output all flow data for what it sees via
JSON messaging allowing you to easily create flow graphs for an address
which has alerted.

On 29/09/17 09:58, Jean-Michel Pouré wrote:
> Why do some guys even **think** about using those old technologies when
> PostgreSQL provides all foundations to implement the same features, but
> in a real database.

On 28/09/17 23:18, Francis Trudeau wrote:
> More generally, what kind of tool except a syslog server to you use to
> study attacks (sorry for this general question) and record traffic in
> a smart way?


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.