[Oisf-users] Problem found // Get from eve.json-> "event_type = file" parser error in elasticsearch
Stefan Sabolowitsch
Stefan.Sabolowitsch at felten-group.com
Thu Feb 13 13:36:08 UTC 2014
Upps sorry,
> btw - your mails are coming awfuly formated - one word per line..?
i hope it’s better now.
can Tom Decanio (or others) make a news git-build or testing ?
Thx for (bug)fixing :)
Stefan
Am 13.02.2014 um 14:28 schrieb Peter Manev <petermanev at gmail.com>:
> On Thu, Feb 13, 2014 at 2:04 PM, Stefan Sabolowitsch
> <Stefan.Sabolowitsch at felten-group.com> wrote:
>> Peter,
>> that is true if you use "only" json log file format, but this elasticsearch
>> machine get tons of log files (firewalls, syslog, event logs etc.) and for
>> that i need the template.
>> Here find you a good explanation why dynamic fields in the particular ".raw"
>> format are important.
>>
>
> oh yes I know why "raw" is important :) and agree :)
> I wasn't aware that you have different inputs, I thought it was only
> the eve.json
>
>
> btw - your mails are coming awfuly formated - one word per line..?
>
>
>> http://www.elasticsearch.org/blog/logstash-1-3-1-released/
>> https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json
>>
>> regards
>> Stefan
>>
>> Am 13.02.2014 um 13:52 schrieb Peter Manev <petermanev at gmail.com>:
>>
>> On Thu, Feb 13, 2014 at 1:42 PM, Stefan Sabolowitsch
>> <Stefan.Sabolowitsch at felten-group.com> wrote:
>>
>> Hi all,
>> yes this is an problem from suri "eve.json file" output format.
>> I change for testing the source code from "output-json-file.c" -> rename
>> "file" to "file_info" and it's work now.
>> An elasticsearch developer said that this is not a bug in elasticsearch, but
>> incorrect json format in particular for dynamic fields.
>> For the reason, i close now the ticket on elasticsearch.
>>
>> https://github.com/elasticsearch/elasticsearch/issues/5084
>>
>> Please suri dev's, change this output format from "eve.json file"
>>
>>
>> I see on the ticket on elastic search you use a template. Why? If you
>> are using the regular eve.json file - you do not need a template to
>> import it to elasticsearch.
>>
>>
>>
>>
>> Thx
>> Stefan
>>
>>
>> "tags" => [],
>> "@version" => 1,
>> "@timestamp" => "2014-02-13T13:22:38.391+01:00",
>> "host" => "ipd1.felten-group.com",
>> "file" => "/nsm/sensor_data/Serrig-intern/eve.json",
>> "message" =>
>> "{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\",\"hostname\":\"ssl.xplosion.de\",\"http_refer\":\"http:\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\",\"http_user_agent\":\"Mozilla\\/5.0
>> (compatible; MSIE 9.0; Windows NT 6.1;
>> Trident\\/5.0)\"},\"file_info\":{\"filename\":\"\\/config\\/douglas.de.config.jsonp\",\"magic\":\"ASCII
>> text, with no line
>> terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}",
>> "type" => "suricata",
>> "received_at" => "2014-02-13 13:22:38 +0100",
>> "event_type" => "file_info",
>> "src_ip" => "205.185.208.58",
>> "src_port" => 80,
>> "proto" => "TCP",
>> "http" => {
>> "url" =>
>> "/config/douglas.de.config.jsonp?cachebuster=234886376939211",
>> "hostname" => "ssl.xplosion.de",
>> "http_refer" =>
>> "http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=",
>> "http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
>> 6.1; Trident/5.0)"
>> },
>> "file_info" => {
>> "filename" => "/config/douglas.de.config.jsonp",
>> "magic" => "ASCII text, with no line terminators",
>> "state" => "CLOSED",
>> "stored" => false,
>> "size" => 230
>> },
>> "dst_ip" => "192.168.1.104",
>> "dst_port" => 52425,
>> "geoip" => {
>> "ip" => "205.185.208.58",
>> "country_code2" => "US",
>> "country_code3" => "USA",
>> "country_name" => "United States",
>> "continent_code" => "NA",
>> "region_name" => "AZ",
>> "city_name" => "Phoenix",
>> "postal_code" => "85012",
>> "latitude" => 33.50829999999999,
>> "longitude" => -112.0717,
>> "dma_code" => 753,
>> "area_code" => 602,
>> "timezone" => "America/Phoenix",
>> "real_region_name" => "Arizona",
>> "location" => [
>> [0] -112.0717,
>> [1] 33.50829999999999
>> ]
>> }
>> }
>>
>>
>> Am 12.02.2014 um 10:03 schrieb Eric Leblond <eric at regit.org>:
>>
>> Hi,
>>
>> On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:
>>
>> Hi all,
>> Get from eve.json-> "event_type = file" parser error in elasticsearch.
>> https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ
>>
>> All other event types work without problem.
>> The interesting thing is however, that can be parsing "files json.log"
>> without problem.
>> Has anyone already successfully sent eve.json-> "event_type = file" to
>> elastic search?
>>
>>
>> On a clean logstash installation, eve.json file event are correctly
>> parsed. By clean, I mean that it has only seen eve.json events.
>>
>> You may have a conflict in elasticsearch because you have two format for
>> file events. I've seen that type of problem once when one of my student
>> did change the type of a key in the output. Injecting of the events did
>> fail after that.
>>
>> If this problem is confirmed, we should maybe do something on code or
>> documentation side to fix this or describe how to fix this in
>> elasticsearch.
>>
>> BR,
>> --
>> Eric Leblond <eric at regit.org>
>>
>>
>>
>>
>>
>>
>> --
>> Regards,
>> Peter Manev
>>
>>
>
>
>
> --
> Regards,
> Peter Manev
>
More information about the Oisf-users
mailing list