[Oisf-users] Problem found // Get from eve.json-> "event_type = file" parser error in elasticsearch

[Stefan Sabolowitsch]

Peter,
that is true if you use „only“ json log file format, but this elasticsearch machine get tons of log files (firewalls, syslog, event logs etc.) and for that i need the template.
Here find you a good explanation why dynamic fields in the particular „.raw" format are important.

http://www.elasticsearch.org/blog/logstash-1-3-1-released/
https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json

regards
Stefan

Am 13.02.2014 um 13:52 schrieb Peter Manev <petermanev at gmail.com<mailto:petermanev at gmail.com>>:

On Thu, Feb 13, 2014 at 1:42 PM, Stefan Sabolowitsch
<Stefan.Sabolowitsch at felten-group.com<mailto:Stefan.Sabolowitsch at felten-group.com>> wrote:
Hi all,
yes this is an problem from suri "eve.json file" output format.
I change for testing the source code from "output-json-file.c" -> rename
"file" to "file_info" and it's work now.
An elasticsearch developer said that this is not a bug in elasticsearch, but
incorrect json format in particular for dynamic fields.
For the reason, i close now the ticket on elasticsearch.

https://github.com/elasticsearch/elasticsearch/issues/5084

Please suri dev's, change this output format from "eve.json file"

I see on the ticket on elastic search you use a template. Why? If you
are using the regular eve.json file - you do not need a template to
import it to elasticsearch.




Thx
Stefan


          "tags" => [],
      "@version" => 1,
    "@timestamp" => "2014-02-13T13:22:38.391+01:00",
          "host" => "ipd1.felten-group.com<http://ipd1.felten-group.com>",
          "file" => "/nsm/sensor_data/Serrig-intern/eve.json",
       "message" =>
"{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\<smb:///config///douglas.de.config.jsonp?cachebuster=234886376939211/>",\"hostname\":\"ssl.xplosion.de<http://ssl.xplosion.de>\",\"http_refer\":\"http:\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\<smb://////ssl.xplosion.de///profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=/>",\"http_user_agent\":\"Mozilla\\/5.0
(compatible; MSIE 9.0; Windows NT 6.1;
Trident\\/5.0)\"},\"file_info\":{\"filename\":\"\\/config\\/douglas.de.config.jsonp\<smb:///config///douglas.de.config.jsonp/>",\"magic\":\"ASCII
text, with no line
terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}",
          "type" => "suricata",
   "received_at" => "2014-02-13 13:22:38 +0100",
    "event_type" => "file_info",
        "src_ip" => "205.185.208.58",
      "src_port" => 80,
         "proto" => "TCP",
          "http" => {
                   "url" =>
"/config/douglas.de.config.jsonp?cachebuster=234886376939211",
              "hostname" => "ssl.xplosion.de<http://ssl.xplosion.de>",
            "http_refer" =>
"http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=",
       "http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT
6.1; Trident/5.0)"
   },
     "file_info" => {
       "filename" => "/config/douglas.de.config.jsonp",
          "magic" => "ASCII text, with no line terminators",
          "state" => "CLOSED",
         "stored" => false,
           "size" => 230
   },
        "dst_ip" => "192.168.1.104",
      "dst_port" => 52425,
         "geoip" => {
                     "ip" => "205.185.208.58",
          "country_code2" => "US",
          "country_code3" => "USA",
           "country_name" => "United States",
         "continent_code" => "NA",
            "region_name" => "AZ",
              "city_name" => "Phoenix",
            "postal_code" => "85012",
               "latitude" => 33.50829999999999,
              "longitude" => -112.0717,
               "dma_code" => 753,
              "area_code" => 602,
               "timezone" => "America/Phoenix",
       "real_region_name" => "Arizona",
               "location" => [
           [0] -112.0717,
           [1] 33.50829999999999
       ]
   }
}


Am 12.02.2014 um 10:03 schrieb Eric Leblond <eric at regit.org<mailto:eric at regit.org>>:

Hi,

On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:

Hi all,
Get from eve.json-> "event_type = file" parser error in elasticsearch.
https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ

All other event types work without problem.
The interesting thing is however, that can be parsing "files json.log"
without problem.
Has anyone already successfully sent eve.json-> "event_type = file" to
elastic search?


On a clean logstash installation, eve.json file event are correctly
parsed. By clean, I mean that it has only seen eve.json events.

You may have a conflict in elasticsearch because you have two format for
file events. I've seen that type of problem once when one of my student
did change the type of a key in the output. Injecting of the events did
fail after that.

If this problem is confirmed, we should maybe do something on code or
documentation side to fix this or describe how to fix this in
elasticsearch.

BR,
--
Eric Leblond <eric at regit.org>






--
Regards,
Peter Manev


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20140213/7cea805c/attachment-0002.html>