<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Peter,</div>
<div><span id="Dst[0][0:2:0:3]" _mstsrc="0_0:2" _mstdst="0_0:3" class="" style="white-space: pre-wrap;">that</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][4:6:5:6]" _mstsrc="0_4:6" _mstdst="0_5:6" class="" style="white-space: pre-wrap;">is</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][8:14:8:11]" _mstsrc="0_8:14" _mstdst="0_8:11" class="" style="white-space: pre-wrap;">true if you use „only“ json log file format</span><span style="white-space: pre-wrap;">,</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][17:20:14:16]" _mstsrc="0_17:20" _mstdst="0_14:16" class="" style="white-space: pre-wrap;">but</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][25:30:18:21]" _mstsrc="0_25:30" _mstdst="0_18:21" style="white-space: pre-wrap;">this</span><span style="white-space: pre-wrap;"> elasticsearch</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][34:40:25:31]" _mstsrc="0_34:40" _mstdst="0_25:31" style="white-space: pre-wrap;">machine</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][42:47:33:35]" _mstsrc="0_42:47" _mstdst="0_33:35" style="white-space: pre-wrap;">get</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][49:54:37:40]" _mstsrc="0_49:54" _mstdst="0_37:40" style="white-space: pre-wrap;">tons</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][56:57:42:43]" _mstsrc="0_56:57" _mstdst="0_42:43" style="white-space: pre-wrap;">of</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][59:66:45:52]" _mstsrc="0_59:66" _mstdst="0_45:52" style="white-space: pre-wrap;">log files (firewalls, syslog, event logs etc.)
</span><span id="Dst[0][71:73:57:59]" _mstsrc="0_71:73" _mstdst="0_57:59" style="white-space: pre-wrap;">and</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][75:79:61:68]" _mstsrc="0_75:79" _mstdst="0_61:68" class="" style="white-space: pre-wrap;">for that</span><span style="white-space: pre-wrap;"> i</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][81:88:72:75]" _mstsrc="0_81:88" _mstdst="0_72:75" class="" style="white-space: pre-wrap;">need</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][94:96:77:79]" _mstsrc="0_94:96" _mstdst="0_77:79" style="white-space: pre-wrap;">the template.</span></div>
<div><span id="Dst[0][0:3:0:3]" _mstsrc="0_0:3" _mstdst="0_0:3" class="" style="white-space: pre-wrap;">Here</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][5:10:5:8]" _mstsrc="0_5:10" _mstdst="0_5:8" class="" style="white-space: pre-wrap;">find</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][12:13:10:12]" _mstsrc="0_12:13" _mstdst="0_10:12" class="" style="white-space: pre-wrap;">you</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][15:17:14:14]" _mstsrc="0_15:17" _mstdst="0_14:14" style="white-space: pre-wrap;">a</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][19:22:16:19]" _mstsrc="0_19:22" _mstdst="0_16:19" class="" style="white-space: pre-wrap;">good</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][24:32:21:31]" _mstsrc="0_24:32" _mstdst="0_21:31" class="" style="white-space: pre-wrap;">explanation</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][34:38:33:35]" _mstsrc="0_34:38" _mstdst="0_33:35" class="" style="white-space: pre-wrap;">why</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][40:49:37:43]" _mstsrc="0_40:49" _mstdst="0_37:43" class="" style="white-space: pre-wrap;">dynamic</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][51:56:45:50]" _mstsrc="0_51:56" _mstdst="0_45:50" style="white-space: pre-wrap;">fields</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][58:59:52:53]" _mstsrc="0_58:59" _mstdst="0_52:53" class="" style="white-space: pre-wrap;">in</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][61:65:55:57]" _mstsrc="0_61:65" _mstdst="0_55:57" style="white-space: pre-wrap;">the</span><span style="white-space: pre-wrap;"> </span>particular „<span style="white-space: pre-wrap;">.</span><span id="Dst[0][67:69:65:67]" _mstsrc="0_67:69" _mstdst="0_65:67" class="" style="white-space: pre-wrap;">raw"</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][71:76:69:74]" _mstsrc="0_71:76" _mstdst="0_69:74" style="white-space: pre-wrap;">format</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][86:89:76:78]" _mstsrc="0_86:89" _mstdst="0_76:78" style="white-space: pre-wrap;">are</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][78:84:80:88]" _mstsrc="0_78:84" _mstdst="0_80:88" class="" style="white-space: pre-wrap;">important</span><span style="white-space: pre-wrap;">.</span></div>
<div><br>
</div>
<div><a href="http://www.elasticsearch.org/blog/logstash-1-3-1-released/">http://www.elasticsearch.org/blog/logstash-1-3-1-released/</a></div>
<div><a href="https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json">https://github.com/logstash/logstash/blob/v1.3.1/lib/logstash/outputs/elasticsearch/elasticsearch-template.json</a></div>
<div><br>
</div>
<div>regards</div>
<div>Stefan</div>
<div><br>
</div>
<div>
<div>Am 13.02.2014 um 13:52 schrieb Peter Manev <<a href="mailto:petermanev@gmail.com">petermanev@gmail.com</a>>:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">On Thu, Feb 13, 2014 at 1:42 PM, Stefan Sabolowitsch<br>
<<a href="mailto:Stefan.Sabolowitsch@felten-group.com">Stefan.Sabolowitsch@felten-group.com</a>> wrote:<br>
<blockquote type="cite">Hi all,<br>
yes this is an problem from suri "eve.json file" output format.<br>
I change for testing the source code from "output-json-file.c" -> rename<br>
"file" to "file_info" and it's work now.<br>
An elasticsearch developer said that this is not a bug in elasticsearch, but<br>
incorrect json format in particular for dynamic fields.<br>
For the reason, i close now the ticket on elasticsearch.<br>
<br>
<a href="https://github.com/elasticsearch/elasticsearch/issues/5084">https://github.com/elasticsearch/elasticsearch/issues/5084</a><br>
<br>
Please suri dev's, change this output format from "eve.json file"<br>
</blockquote>
<br>
I see on the ticket on elastic search you use a template. Why? If you<br>
are using the regular eve.json file - you do not need a template to<br>
import it to elasticsearch.<br>
<br>
<br>
<br>
<blockquote type="cite"><br>
Thx<br>
Stefan<br>
<br>
<br>
"tags" => [],<br>
"@version" => 1,<br>
"@timestamp" => "2014-02-13T13:22:38.391+01:00",<br>
"host" => "<a href="http://ipd1.felten-group.com">ipd1.felten-group.com</a>",<br>
"file" => "/nsm/sensor_data/Serrig-intern/eve.json",<br>
"message" =><br>
"{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"<a href="smb:///config///douglas.de.config.jsonp?cachebuster=234886376939211/">\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\</a>",\"hostname\":\"<a href="http://ssl.xplosion.de">ssl.xplosion.de</a>\",\"http_refer\":\"http:<a href="smb://////ssl.xplosion.de///profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=/">\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\</a>",\"http_user_agent\":\"Mozilla\\/5.0<br>
(compatible; MSIE 9.0; Windows NT 6.1;<br>
Trident\\/5.0)\"},\"file_info\":{\"filename\":\"<a href="smb:///config///douglas.de.config.jsonp/">\\/config\\/douglas.de.config.jsonp\</a>",\"magic\":\"ASCII<br>
text, with no line<br>
terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}",<br>
"type" => "suricata",<br>
"received_at" => "2014-02-13 13:22:38 +0100",<br>
"event_type" => "file_info",<br>
"src_ip" => "205.185.208.58",<br>
"src_port" => 80,<br>
"proto" => "TCP",<br>
"http" => {<br>
"url" =><br>
"/config/douglas.de.config.jsonp?cachebuster=234886376939211",<br>
"hostname" => "<a href="http://ssl.xplosion.de">ssl.xplosion.de</a>",<br>
"http_refer" =><br>
"<a href="http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=">http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=</a>",<br>
"http_user_agent" => "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT<br>
6.1; Trident/5.0)"<br>
},<br>
"file_info" => {<br>
"filename" => "/config/douglas.de.config.jsonp",<br>
"magic" => "ASCII text, with no line terminators",<br>
"state" => "CLOSED",<br>
"stored" => false,<br>
"size" => 230<br>
},<br>
"dst_ip" => "192.168.1.104",<br>
"dst_port" => 52425,<br>
"geoip" => {<br>
"ip" => "205.185.208.58",<br>
"country_code2" => "US",<br>
"country_code3" => "USA",<br>
"country_name" => "United States",<br>
"continent_code" => "NA",<br>
"region_name" => "AZ",<br>
"city_name" => "Phoenix",<br>
"postal_code" => "85012",<br>
"latitude" => 33.50829999999999,<br>
"longitude" => -112.0717,<br>
"dma_code" => 753,<br>
"area_code" => 602,<br>
"timezone" => "America/Phoenix",<br>
"real_region_name" => "Arizona",<br>
"location" => [<br>
[0] -112.0717,<br>
[1] 33.50829999999999<br>
]<br>
}<br>
}<br>
<br>
<br>
Am 12.02.2014 um 10:03 schrieb Eric Leblond <<a href="mailto:eric@regit.org">eric@regit.org</a>>:<br>
<br>
Hi,<br>
<br>
On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:<br>
<br>
Hi all,<br>
Get from eve.json-> "event_type = file" parser error in elasticsearch.<br>
<a href="https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ">https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ</a><br>
<br>
All other event types work without problem.<br>
The interesting thing is however, that can be parsing "files json.log"<br>
without problem.<br>
Has anyone already successfully sent eve.json-> "event_type = file" to<br>
elastic search?<br>
<br>
<br>
On a clean logstash installation, eve.json file event are correctly<br>
parsed. By clean, I mean that it has only seen eve.json events.<br>
<br>
You may have a conflict in elasticsearch because you have two format for<br>
file events. I've seen that type of problem once when one of my student<br>
did change the type of a key in the output. Injecting of the events did<br>
fail after that.<br>
<br>
If this problem is confirmed, we should maybe do something on code or<br>
documentation side to fix this or describe how to fix this in<br>
elasticsearch.<br>
<br>
BR,<br>
--<br>
Eric Leblond <eric@regit.org><br>
<br>
<br>
<br>
</blockquote>
<br>
<br>
<br>
-- <br>
Regards,<br>
Peter Manev<br>
<br>
</blockquote>
</div>
<br>
</body>
</html>