[Oisf-users] Problem found // Get from eve.json-> "event_type = file" parser error in elasticsearch

[Peter Manev]

On Thu, Feb 13, 2014 at 2:04 PM, Stefan Sabolowitsch
<Stefan.Sabolowitsch at felten-group.com> wrote:
> Peter,
> that is true if you use "only" json log file format, but this elasticsearch
> machine get tons of log files (firewalls, syslog, event logs etc.) and for
> that i need the template.
> Here find you a good explanation why dynamic fields in the particular ".raw"
> format are important.
>

oh yes I know  why "raw" is important :) and agree :)
I wasn't aware that you have different inputs, I  thought it was only
the eve.json


btw - your mails are coming awfuly formated - one word per line..?


> http://www.elasticsearch.org/blog/logstash-1-3-1-released/
> https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json
>
> regards
> Stefan
>
> Am 13.02.2014 um 13:52 schrieb Peter Manev <petermanev at gmail.com>:
>
> On Thu, Feb 13, 2014 at 1:42 PM, Stefan Sabolowitsch
> <Stefan.Sabolowitsch at felten-group.com> wrote:
>
> Hi all,
> yes this is an problem from suri "eve.json file" output format.
> I change for testing the source code from "output-json-file.c" -> rename
> "file" to "file_info" and it's work now.
> An elasticsearch developer said that this is not a bug in elasticsearch, but
> incorrect json format in particular for dynamic fields.
> For the reason, i close now the ticket on elasticsearch.
>
> https://github.com/elasticsearch/elasticsearch/issues/5084
>
> Please suri dev's, change this output format from "eve.json file"
>
>
> I see on the ticket on elastic search you use a template. Why? If you
> are using the regular eve.json file - you do not need a template to
> import it to elasticsearch.
>
>
>
>
> Thx
> Stefan
>
>
>           "tags" => [],
>       "@version" => 1,
>     "@timestamp" => "2014-02-13T13:22:38.391+01:00",
>           "host" => "ipd1.felten-group.com",
>           "file" => "/nsm/sensor_data/Serrig-intern/eve.json",
>        "message" =>
> "{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\",\"hostname\":\"ssl.xplosion.de\",\"http_refer\":\"http:\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\",\"http_user_agent\":\"Mozilla\\/5.0
> (compatible; MSIE 9.0; Windows NT 6.1;
> Trident\\/5.0)\"},\"file_info\":{\"filename\":\"\\/config\\/douglas.de.config.jsonp\",\"magic\":\"ASCII
> text, with no line
> terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}",
>           "type" => "suricata",
>    "received_at" => "2014-02-13 13:22:38 +0100",
>     "event_type" => "file_info",
>         "src_ip" => "205.185.208.58",
>       "src_port" => 80,
>          "proto" => "TCP",
>           "http" => {
>                    "url" =>
> "/config/douglas.de.config.jsonp?cachebuster=234886376939211",
>               "hostname" => "ssl.xplosion.de",
>             "http_refer" =>
> "http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=",
>        "http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
> 6.1; Trident/5.0)"
>    },
>      "file_info" => {
>        "filename" => "/config/douglas.de.config.jsonp",
>           "magic" => "ASCII text, with no line terminators",
>           "state" => "CLOSED",
>          "stored" => false,
>            "size" => 230
>    },
>         "dst_ip" => "192.168.1.104",
>       "dst_port" => 52425,
>          "geoip" => {
>                      "ip" => "205.185.208.58",
>           "country_code2" => "US",
>           "country_code3" => "USA",
>            "country_name" => "United States",
>          "continent_code" => "NA",
>             "region_name" => "AZ",
>               "city_name" => "Phoenix",
>             "postal_code" => "85012",
>                "latitude" => 33.50829999999999,
>               "longitude" => -112.0717,
>                "dma_code" => 753,
>               "area_code" => 602,
>                "timezone" => "America/Phoenix",
>        "real_region_name" => "Arizona",
>                "location" => [
>            [0] -112.0717,
>            [1] 33.50829999999999
>        ]
>    }
> }
>
>
> Am 12.02.2014 um 10:03 schrieb Eric Leblond <eric at regit.org>:
>
> Hi,
>
> On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:
>
> Hi all,
> Get from eve.json-> "event_type = file" parser error in elasticsearch.
> https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ
>
> All other event types work without problem.
> The interesting thing is however, that can be parsing "files json.log"
> without problem.
> Has anyone already successfully sent eve.json-> "event_type = file" to
> elastic search?
>
>
> On a clean logstash installation, eve.json file event are correctly
> parsed. By clean, I mean that it has only seen eve.json events.
>
> You may have a conflict in elasticsearch because you have two format for
> file events. I've seen that type of problem once when one of my student
> did change the type of a key in the output. Injecting of the events did
> fail after that.
>
> If this problem is confirmed, we should maybe do something on code or
> documentation side to fix this or describe how to fix this in
> elasticsearch.
>
> BR,
> --
> Eric Leblond <eric at regit.org>
>
>
>
>
>
>
> --
> Regards,
> Peter Manev
>
>



-- 
Regards,
Peter Manev