[Oisf-users] Problem found // Get from eve.json-> "event_type = file" parser error in elasticsearch

[Peter Manev]

On Thu, Feb 13, 2014 at 1:42 PM, Stefan Sabolowitsch
<Stefan.Sabolowitsch at felten-group.com> wrote:
> Hi all,
> yes this is an problem from suri "eve.json file" output format.
> I change for testing the source code from "output-json-file.c" -> rename
> "file" to "file_info" and it's work now.
> An elasticsearch developer said that this is not a bug in elasticsearch, but
> incorrect json format in particular for dynamic fields.
> For the reason, i close now the ticket on elasticsearch.
>
> https://github.com/elasticsearch/elasticsearch/issues/5084
>
> Please suri dev's, change this output format from "eve.json file"

I see on the ticket on elastic search you use a template. Why? If you
are using the regular eve.json file - you do not need a template to
import it to elasticsearch.



>
> Thx
> Stefan
>
>
>            "tags" => [],
>        "@version" => 1,
>      "@timestamp" => "2014-02-13T13:22:38.391+01:00",
>            "host" => "ipd1.felten-group.com",
>            "file" => "/nsm/sensor_data/Serrig-intern/eve.json",
>         "message" =>
> "{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\",\"hostname\":\"ssl.xplosion.de\",\"http_refer\":\"http:\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\",\"http_user_agent\":\"Mozilla\\/5.0
> (compatible; MSIE 9.0; Windows NT 6.1;
> Trident\\/5.0)\"},\"file_info\":{\"filename\":\"\\/config\\/douglas.de.config.jsonp\",\"magic\":\"ASCII
> text, with no line
> terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}",
>            "type" => "suricata",
>     "received_at" => "2014-02-13 13:22:38 +0100",
>      "event_type" => "file_info",
>          "src_ip" => "205.185.208.58",
>        "src_port" => 80,
>           "proto" => "TCP",
>            "http" => {
>                     "url" =>
> "/config/douglas.de.config.jsonp?cachebuster=234886376939211",
>                "hostname" => "ssl.xplosion.de",
>              "http_refer" =>
> "http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=",
>         "http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
> 6.1; Trident/5.0)"
>     },
>       "file_info" => {
>         "filename" => "/config/douglas.de.config.jsonp",
>            "magic" => "ASCII text, with no line terminators",
>            "state" => "CLOSED",
>           "stored" => false,
>             "size" => 230
>     },
>          "dst_ip" => "192.168.1.104",
>        "dst_port" => 52425,
>           "geoip" => {
>                       "ip" => "205.185.208.58",
>            "country_code2" => "US",
>            "country_code3" => "USA",
>             "country_name" => "United States",
>           "continent_code" => "NA",
>              "region_name" => "AZ",
>                "city_name" => "Phoenix",
>              "postal_code" => "85012",
>                 "latitude" => 33.50829999999999,
>                "longitude" => -112.0717,
>                 "dma_code" => 753,
>                "area_code" => 602,
>                 "timezone" => "America/Phoenix",
>         "real_region_name" => "Arizona",
>                 "location" => [
>             [0] -112.0717,
>             [1] 33.50829999999999
>         ]
>     }
> }
>
>
> Am 12.02.2014 um 10:03 schrieb Eric Leblond <eric at regit.org>:
>
> Hi,
>
> On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:
>
> Hi all,
> Get from eve.json-> "event_type = file" parser error in elasticsearch.
> https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ
>
> All other event types work without problem.
> The interesting thing is however, that can be parsing "files json.log"
> without problem.
> Has anyone already successfully sent eve.json-> "event_type = file" to
> elastic search?
>
>
> On a clean logstash installation, eve.json file event are correctly
> parsed. By clean, I mean that it has only seen eve.json events.
>
> You may have a conflict in elasticsearch because you have two format for
> file events. I've seen that type of problem once when one of my student
> did change the type of a key in the output. Injecting of the events did
> fail after that.
>
> If this problem is confirmed, we should maybe do something on code or
> documentation side to fix this or describe how to fix this in
> elasticsearch.
>
> BR,
> --
> Eric Leblond <eric at regit.org>
>
>
>



-- 
Regards,
Peter Manev