[Oisf-users] Problem found // Get from eve.json-> "event_type = file" parser error in elasticsearch

Stefan Sabolowitsch Stefan.Sabolowitsch at felten-group.com
Thu Feb 13 13:04:44 UTC 2014

that is true if you use „only“ json log file format, but this elasticsearch machine get tons of log files (firewalls, syslog, event logs etc.) and for that i need the template.
Here find you a good explanation why dynamic fields in the particular „.raw" format are important.



Am 13.02.2014 um 13:52 schrieb Peter Manev <petermanev at gmail.com<mailto:petermanev at gmail.com>>:

On Thu, Feb 13, 2014 at 1:42 PM, Stefan Sabolowitsch
<Stefan.Sabolowitsch at felten-group.com<mailto:Stefan.Sabolowitsch at felten-group.com>> wrote:
Hi all,
yes this is an problem from suri "eve.json file" output format.
I change for testing the source code from "output-json-file.c" -> rename
"file" to "file_info" and it's work now.
An elasticsearch developer said that this is not a bug in elasticsearch, but
incorrect json format in particular for dynamic fields.
For the reason, i close now the ticket on elasticsearch.


Please suri dev's, change this output format from "eve.json file"

I see on the ticket on elastic search you use a template. Why? If you
are using the regular eve.json file - you do not need a template to
import it to elasticsearch.


          "tags" => [],
      "@version" => 1,
    "@timestamp" => "2014-02-13T13:22:38.391+01:00",
          "host" => "ipd1.felten-group.com<http://ipd1.felten-group.com>",
          "file" => "/nsm/sensor_data/Serrig-intern/eve.json",
       "message" =>
(compatible; MSIE 9.0; Windows NT 6.1;
text, with no line
          "type" => "suricata",
   "received_at" => "2014-02-13 13:22:38 +0100",
    "event_type" => "file_info",
        "src_ip" => "",
      "src_port" => 80,
         "proto" => "TCP",
          "http" => {
                   "url" =>
              "hostname" => "ssl.xplosion.de<http://ssl.xplosion.de>",
            "http_refer" =>
       "http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
6.1; Trident/5.0)"
     "file_info" => {
       "filename" => "/config/douglas.de.config.jsonp",
          "magic" => "ASCII text, with no line terminators",
          "state" => "CLOSED",
         "stored" => false,
           "size" => 230
        "dst_ip" => "",
      "dst_port" => 52425,
         "geoip" => {
                     "ip" => "",
          "country_code2" => "US",
          "country_code3" => "USA",
           "country_name" => "United States",
         "continent_code" => "NA",
            "region_name" => "AZ",
              "city_name" => "Phoenix",
            "postal_code" => "85012",
               "latitude" => 33.50829999999999,
              "longitude" => -112.0717,
               "dma_code" => 753,
              "area_code" => 602,
               "timezone" => "America/Phoenix",
       "real_region_name" => "Arizona",
               "location" => [
           [0] -112.0717,
           [1] 33.50829999999999

Am 12.02.2014 um 10:03 schrieb Eric Leblond <eric at regit.org<mailto:eric at regit.org>>:


On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:

Hi all,
Get from eve.json-> "event_type = file" parser error in elasticsearch.

All other event types work without problem.
The interesting thing is however, that can be parsing "files json.log"
without problem.
Has anyone already successfully sent eve.json-> "event_type = file" to
elastic search?

On a clean logstash installation, eve.json file event are correctly
parsed. By clean, I mean that it has only seen eve.json events.

You may have a conflict in elasticsearch because you have two format for
file events. I've seen that type of problem once when one of my student
did change the type of a key in the output. Injecting of the events did
fail after that.

If this problem is confirmed, we should maybe do something on code or
documentation side to fix this or describe how to fix this in

Eric Leblond <eric at regit.org>

Peter Manev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140213/7cea805c/attachment-0002.html>

More information about the Oisf-users mailing list