<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>Hi all,</div>
<div>yes this is an problem from suri "eve.json file" output format.</div>
<div>I change for testing the source code from "output-json-file.c“ -> rename „file" to „file_info" and it’s work now.</div>
<div><span id="Dst[0][0:2:0:1]" _mstsrc="0_0:2" _mstdst="0_0:1" class="" style="white-space: pre-wrap;">An</span><span style="white-space: pre-wrap;"> elasticsearch</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][6:15:5:13]" _mstsrc="0_6:15" _mstdst="0_5:13" style="white-space: pre-wrap;">developer</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][17:21:15:18]" _mstsrc="0_17:21" _mstdst="0_15:18" style="white-space: pre-wrap;">said</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][23:25:20:23]" _mstsrc="0_23:25" _mstdst="0_20:23" style="white-space: pre-wrap;">that</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][27:30:25:28]" _mstsrc="0_27:30" _mstdst="0_25:28" style="white-space: pre-wrap;">this</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][46:49:30:31]" _mstsrc="0_46:49" _mstdst="0_30:31" style="white-space: pre-wrap;">is</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][32:35:33:37]" _mstsrc="0_32:35" _mstdst="0_33:37" style="white-space: pre-wrap;">not a</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][37:39:39:41]" _mstsrc="0_37:39" _mstdst="0_39:41" style="white-space: pre-wrap;">bug</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][41:42:43:44]" _mstsrc="0_41:42" _mstdst="0_43:44" style="white-space: pre-wrap;">in</span><span style="white-space: pre-wrap;"> </span><span style="white-space: pre-wrap;">elasticsearch</span><span style="white-space: pre-wrap;">,</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][52:58:49:51]" _mstsrc="0_52:58" _mstdst="0_49:51" style="white-space: pre-wrap;">but</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][60:67:53:61]" _mstsrc="0_60:67" _mstdst="0_53:61" style="white-space: pre-wrap;">incorrect</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][69:72:63:66]" _mstsrc="0_69:72" _mstdst="0_63:66" style="white-space: pre-wrap;">json</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][74:79:68:73]" _mstsrc="0_74:79" _mstdst="0_68:73" class="" style="white-space: pre-wrap;">format</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][81:82:75:76]" _mstsrc="0_81:82" _mstdst="0_75:76" style="white-space: pre-wrap;">in</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][95:97:78:87]" _mstsrc="0_95:97" _mstdst="0_78:87" style="white-space: pre-wrap;">particular</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][99:108:89:91]" _mstsrc="0_99:108" _mstdst="0_89:91" class="" style="white-space: pre-wrap;">for</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][110:115:93:99]" _mstsrc="0_110:115" _mstdst="0_93:99" class="" style="white-space: pre-wrap;">dynamic</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][116:116:101:106]" _mstsrc="0_116:116" _mstdst="0_101:106" class="" style="white-space: pre-wrap;">fields</span><span style="white-space: pre-wrap;">.</span> </div>
<div><span id="Dst[0][0:2:0:2]" _mstsrc="0_0:2" _mstdst="0_0:2" class="" style="white-space: pre-wrap;">For</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][4:6:4:6]" _mstsrc="0_4:6" _mstdst="0_4:6" class="" style="white-space: pre-wrap;">the</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][8:12:8:13]" _mstsrc="0_8:12" _mstdst="0_8:13" class="" style="white-space: pre-wrap;">reason</span><span style="white-space: pre-wrap;">,</span><span style="white-space: pre-wrap;"> i</span><span style="white-space: pre-wrap;"> close</span><span style="white-space: pre-wrap;">
now </span><span id="Dst[0][28:30:29:31]" _mstsrc="0_28:30" _mstdst="0_29:31" class="" style="white-space: pre-wrap;">the</span><span style="white-space: pre-wrap;">
</span><span id="Dst[0][32:37:33:38]" _mstsrc="0_32:37" _mstdst="0_33:38" class="" style="white-space: pre-wrap;">ticket</span><span style="white-space: pre-wrap;"> on elasticsearch.</span></div>
<div><span style="white-space: pre-wrap;"><br>
</span></div>
<div><a href="https://github.com/elasticsearch/elasticsearch/issues/5084">https://github.com/elasticsearch/elasticsearch/issues/5084</a></div>
<div><br>
</div>
<div>Please suri dev’s, change this output format from "eve.json file"</div>
<div><span style="white-space: pre-wrap;"><br>
</span></div>
<div><span style="white-space: pre-wrap;">Thx</span></div>
<div><span style="white-space: pre-wrap;">Stefan</span></div>
<div><br>
</div>
<div><br>
</div>
<div>
<div style="margin: 0px; font-size: 12px; font-family: Menlo;">
<div style="margin: 0px;"> "tags"<span style="color: #cbcbcb"> => </span>
[],</div>
<div style="margin: 0px;"> "@version"<span style="color: #cbcbcb"> => </span>
<span style="color: #5330e1"><b>1</b></span>,</div>
<div style="margin: 0px; color: rgb(52, 189, 38);"><span style="color: #000000"> "@timestamp"</span><span style="color: #cbcbcb"> =>
</span>"2014-02-13T13:22:38.391+01:00"<span style="color: #000000">,</span></div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "host"</span><span style="color: #cbcbcb"> =>
</span>"<a href="http://ipd1.felten-group.com">ipd1.felten-group.com</a>"<span style="color: #000000">,</span></div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "file"</span><span style="color: #cbcbcb"> =>
</span>"/nsm/sensor_data/Serrig-intern/eve.json"<span style="color: #000000">,</span></div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "message"</span><span style="color: #cbcbcb"> =>
</span>"{\"time\":\"02\\/13\\/2014-12:22:38.391825\",\"event_type\":\"file_info\",\"src_ip\":\"205.185.208.58\",\"src_port\":80,\"dest_ip\":\"192.168.1.104\",\"dest_port\":52425,\"proto\":\"TCP\",\"http\":{\"url\":\"<a href="smb:///config///douglas.de.config.jsonp?cachebuster=234886376939211/">\\/config\\/douglas.de.config.jsonp?cachebuster=234886376939211\</a>",\"hostname\":\"<a href="http://ssl.xplosion.de">ssl.xplosion.de</a>\",\"http_refer\":\"http:<a href="smb://////ssl.xplosion.de///profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=/">\\/\\/ssl.xplosion.de\\/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=\</a>",\"http_user_agent\":\"Mozilla\\/5.0
(compatible; MSIE 9.0; Windows NT 6.1; Trident\\/5.0)\"},\"file_info\":{\"filename\":\"<a href="smb:///config///douglas.de.config.jsonp/">\\/config\\/douglas.de.config.jsonp\</a>",\"magic\":\"ASCII text, with no line terminators\",\"state\":\"CLOSED\",\"stored\":false,\"size\":230}}"<span style="color: #000000">,</span></div>
<div style="margin: 0px;"> "type"<span style="color: #cbcbcb"> => </span>
<span style="color: #afad24">"suricata"</span>,</div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "received_at"</span><span style="color: #cbcbcb"> =>
</span>"2014-02-13 13:22:38 +0100"<span style="color: #000000">,</span></div>
<div style="margin: 0px;"> "event_type"<span style="color: #cbcbcb"> => </span>
<span style="color: #afad24">"file_info"</span>,</div>
<div style="margin: 0px;"> "src_ip"<span style="color: #cbcbcb"> => </span>
<span style="color: #afad24">"205.185.208.58"</span>,</div>
<div style="margin: 0px;"> "src_port"<span style="color: #cbcbcb"> => </span>
<span style="color: #5330e1"><b>80</b></span>,</div>
<div style="margin: 0px;"> "proto"<span style="color: #cbcbcb"> => </span>
<span style="color: #afad24">"TCP"</span>,</div>
<div style="margin: 0px;"> "http"<span style="color: #cbcbcb"> => </span>
{</div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "url"</span><span style="color: #cbcbcb"> =>
</span>"/config/douglas.de.config.jsonp?cachebuster=234886376939211"<span style="color: #000000">,</span></div>
<div style="margin: 0px;"> "hostname"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"<a href="http://ssl.xplosion.de">ssl.xplosion.de</a>"</span>,</div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "http_refer"</span><span style="color: #cbcbcb"> =>
</span>"<a href="http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=">http://ssl.xplosion.de/profiler.html?customer=douglas.de&event_id=shop_visit&shop_id=Accessoires%3ESchmuck%3EOhrringe&shop_trackingproducts=</a>"<span style="color: #000000">,</span></div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "http_user_agent"</span><span style="color: #cbcbcb"> =>
</span>"Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)"</div>
<div style="margin: 0px;"> },</div>
<div style="margin: 0px;"> "file_info"<span style="color: #cbcbcb"> => </span>
{</div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "filename"</span><span style="color: #cbcbcb"> =>
</span>"/config/douglas.de.config.jsonp"<span style="color: #000000">,</span></div>
<div style="margin: 0px; color: rgb(175, 173, 36);"><span style="color: #000000"> "magic"</span><span style="color: #cbcbcb"> =>
</span>"ASCII text, with no line terminators"<span style="color: #000000">,</span></div>
<div style="margin: 0px;"> "state"<span style="color: #cbcbcb"> => </span>
<span style="color: #afad24">"CLOSED"</span>,</div>
<div style="margin: 0px;"> "stored"<span style="color: #cbcbcb"> => </span>
<span style="color: #c33720"><b>false</b></span>,</div>
<div style="margin: 0px;"> "size"<span style="color: #cbcbcb"> => </span>
<span style="color: #5330e1"><b>230</b></span></div>
<div style="margin: 0px;"> },</div>
<div style="margin: 0px;"> "dst_ip"<span style="color: #cbcbcb"> => </span>
<span style="color: #afad24">"192.168.1.104"</span>,</div>
<div style="margin: 0px;"> "dst_port"<span style="color: #cbcbcb"> => </span>
<span style="color: #5330e1"><b>52425</b></span>,</div>
<div style="margin: 0px;"> "geoip"<span style="color: #cbcbcb"> => </span>
{</div>
<div style="margin: 0px;"> "ip"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"205.185.208.58"</span>,</div>
<div style="margin: 0px;"> "country_code2"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"US"</span>,</div>
<div style="margin: 0px;"> "country_code3"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"USA"</span>,</div>
<div style="margin: 0px;"> "country_name"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"United States"</span>,</div>
<div style="margin: 0px;"> "continent_code"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"NA"</span>,</div>
<div style="margin: 0px;"> "region_name"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"AZ"</span>,</div>
<div style="margin: 0px;"> "city_name"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"Phoenix"</span>,</div>
<div style="margin: 0px;"> "postal_code"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"85012"</span>,</div>
<div style="margin: 0px;"> "latitude"<span style="color: #cbcbcb"> =>
</span><span style="color: #5330e1"><b>33.50829999999999</b></span>,</div>
<div style="margin: 0px;"> "longitude"<span style="color: #cbcbcb"> =>
</span><span style="color: #5330e1"><b>-112.0717</b></span>,</div>
<div style="margin: 0px;"> "dma_code"<span style="color: #cbcbcb"> =>
</span><span style="color: #5330e1"><b>753</b></span>,</div>
<div style="margin: 0px;"> "area_code"<span style="color: #cbcbcb"> =>
</span><span style="color: #5330e1"><b>602</b></span>,</div>
<div style="margin: 0px;"> "timezone"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"America/Phoenix"</span>,</div>
<div style="margin: 0px;"> "real_region_name"<span style="color: #cbcbcb"> =>
</span><span style="color: #afad24">"Arizona"</span>,</div>
<div style="margin: 0px;"> "location"<span style="color: #cbcbcb"> =>
</span>[</div>
<div style="margin: 0px;"> <span style="color: #cbcbcb"><b>[0] </b></span><span style="color: #5330e1"><b>-112.0717</b></span>,</div>
<div style="margin: 0px; color: rgb(83, 48, 225);"><span style="color: #000000">
</span><span style="color: #cbcbcb"><b>[1] </b></span><b>33.50829999999999</b></div>
<div style="margin: 0px;"> ]</div>
<div style="margin: 0px;"> }</div>
<div style="margin: 0px;">}</div>
<div><br>
</div>
</div>
</div>
<br>
<div>
<div>Am 12.02.2014 um 10:03 schrieb Eric Leblond <<a href="mailto:eric@regit.org">eric@regit.org</a>>:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">Hi,<br>
<br>
On Wed, 2014-02-12 at 08:40 +0000, Stefan Sabolowitsch wrote:<br>
<blockquote type="cite">Hi all,<br>
Get from eve.json-> "event_type = file" parser error in elasticsearch.<br>
<a href="https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ">https://groups.google.com/d/msg/elasticsearch/1P3fM0oa7gU/8g0qqUxfPSoJ</a><br>
<br>
All other event types work without problem.<br>
The interesting thing is however, that can be parsing "files json.log" without problem.<br>
Has anyone already successfully sent eve.json-> "event_type = file“ to elastic search?<br>
</blockquote>
<br>
On a clean logstash installation, eve.json file event are correctly<br>
parsed. By clean, I mean that it has only seen eve.json events.<br>
<br>
You may have a conflict in elasticsearch because you have two format for<br>
file events. I've seen that type of problem once when one of my student<br>
did change the type of a key in the output. Injecting of the events did<br>
fail after that.<br>
<br>
If this problem is confirmed, we should maybe do something on code or<br>
documentation side to fix this or describe how to fix this in<br>
elasticsearch.<br>
<br>
BR,<br>
-- <br>
Eric Leblond <<a href="mailto:eric@regit.org">eric@regit.org</a>><br>
<br>
<br>
</blockquote>
</div>
<br>
</body>
</html>