[Oisf-users] struggling to get eve logs into ES

[Jason Ish]

Hello Russel,

On 2020-05-10 4:25 p.m., Russell Fulton wrote:
> I am attempting to get eve.json data into elasticsearch.
> 
> I found filebeat and noted that it had a suricata plugin.  Installed it
> (following the instruction in the docs). 

The Filebeat Suricata plugin doesn't import Suricata events as-is.
Instead of transforms them into ECS (Elastic Common Schema) and I
believe sets up some SIEM style Dashboards. I haven't played around with
it much, as the event transformation to ECS breaks existing Kibana
dashboards as found in SELKS, and is different enough from raw Suricata
events that EveBox doesn't understand them. So keep that in mind if
EveBox is a goal.  I would like to have EveBox just work against the ECS
formatted Suricata events, and hopefully will do this soon, but nothings
been started yet.

> 
> At first I could not find the index and the doc were not forthcoming
> about the naming.  Eventually I figured out that it was called
> filebeat-…..   By that time the index contained 50GB of data and was
> marked as closed.  
> 
> I can’t do anything with the closed index, even delete it. I have tried
> opening it by sending API request but that comes back with
> “acknowledged” but nothig changes.  Filebeat logs suggest it it still
> sendingdata to ES???
> 
> I tried changing the index name but filebeat now insists that I now need
> to load the templates for the new index without saying how. Re-running
> filebeat —setup does not do it.
> 
> Before spending more time going around in circles I thought I would ask
> what others are doing to get their eve logs into ES so that they can use
> Evebox — which is next on my list of things to look at once I get the
> data loading into ES.

A few options I'm more familiar with are

- Filebeat without the Suricata plugin, but json codec on. This can send
events directly to Elasticsearch, and can be used with EveBox, however
searching is limited to exact string matches. This is due to the
Filebeat index template.

- Just use Logstash with the json codec on. This uses the Logstash
template which analyzes more data and makes search simpler.

- Use Filebeat and Logstash. Run one instance of Logstash somewhere with
the Filebeat input turned on.  Then on your sensors run Filebeat and
have them send to Logstash.  This way you get the lighter weight
Filebeat on your sensors, but the flexibility of Logstash.

- Use EveBox agent (or esimport). This is lightweight, and imports
events to Elasticsearch in a fully compatible Logstash index way.

Hope that helps,
Jason