[Oisf-users] Problem found // Get from eve.json-> "event_type = file" parser error in elasticsearch

[Stefan Sabolowitsch]

Hi all,
yes this is an problem from suri "eve.json file" output format.
I change for testing the source code from "output-json-file.c“ -> rename „file" to „file_info" and it’s work now.
An elasticsearch developer said that this is not a bug in elasticsearch, but incorrect json format in particular for dynamic fields.
For the reason, i close now the ticket on elasticsearch.

https://github.com/elasticsearch/elasticsearch/issues/5084

Please suri dev’s, change this output format from "eve.json file"

Thx
Stefan


           "tags" => [],
       "@version" => 1,
     "@timestamp" => "2014-02-13T13:22:38.391+01:00",
           "host" => "ipd1.felten-group.com<http://ipd1.felten-group.com>",
           "file" => "/nsm/sensor_data/Serrig-intern/eve.json",
        "message" => "{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\<smb:///config///douglas.de.config.jsonp?cachebuster=234886376939211/>",\"hostname\":\"ssl.xplosion.de<http://ssl.xplosion.de>\",\"http_refer\":\"http:\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\<smb://////ssl.xplosion.de///profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=/>",\"http_user_agent\":\"Mozilla\\/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident\\/5.0)\"},\"file_info\":{\"filename\":\"\\/config\\/douglas.de.config.jsonp\<smb:///config///douglas.de.config.jsonp/>",\"magic\":\"ASCII text, with no line terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}",
           "type" => "suricata",
    "received_at" => "2014-02-13 13:22:38 +0100",
     "event_type" => "file_info",
         "src_ip" => "205.185.208.58",
       "src_port" => 80,
          "proto" => "TCP",
           "http" => {
                    "url" => "/config/douglas.de.config.jsonp?cachebuster=234886376939211",
               "hostname" => "ssl.xplosion.de<http://ssl.xplosion.de>",
             "http_refer" => "http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=",
        "http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"
    },
      "file_info" => {
        "filename" => "/config/douglas.de.config.jsonp",
           "magic" => "ASCII text, with no line terminators",
           "state" => "CLOSED",
          "stored" => false,
            "size" => 230
    },
         "dst_ip" => "192.168.1.104",
       "dst_port" => 52425,
          "geoip" => {
                      "ip" => "205.185.208.58",
           "country_code2" => "US",
           "country_code3" => "USA",
            "country_name" => "United States",
          "continent_code" => "NA",
             "region_name" => "AZ",
               "city_name" => "Phoenix",
             "postal_code" => "85012",
                "latitude" => 33.50829999999999,
               "longitude" => -112.0717,
                "dma_code" => 753,
               "area_code" => 602,
                "timezone" => "America/Phoenix",
        "real_region_name" => "Arizona",
                "location" => [
            [0] -112.0717,
            [1] 33.50829999999999
        ]
    }
}


Am 12.02.2014 um 10:03 schrieb Eric Leblond <eric at regit.org<mailto:eric at regit.org>>:

Hi,

On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:
Hi all,
Get from eve.json-> "event_type = file" parser error in elasticsearch.
https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ

All other event types work without problem.
The interesting thing is however, that can be parsing "files json.log" without problem.
Has anyone already successfully sent eve.json-> "event_type = file“ to elastic search?

On a clean logstash installation, eve.json file event are correctly
parsed. By clean, I mean that it has only seen eve.json events.

You may have a conflict in elasticsearch because you have two format for
file events. I've seen that type of problem once when one of my student
did change the type of a key in the output. Injecting of the events did
fail after that.

If this problem is confirmed, we should maybe do something on code or
documentation side to fix this or describe how to fix this in
elasticsearch.

BR,
--
Eric Leblond <eric at regit.org<mailto:eric at regit.org>>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140213/4d8c8680/attachment-0002.html>