[Oisf-users] Exe File Extraction by using Suricata
Peter Manev
petermanev at gmail.com
Mon Mar 17 13:26:36 UTC 2014
On Mon, Mar 17, 2014 at 2:20 PM, Mustafa Çoker <mustafacoker at yandex.com> wrote:
> Hi EveryBody,
> I am new to Suricata.
> I want to exctract executable files by using Suricata File extraction rule
> sample. (files.rules)
> I am using Ubuntu 13.10 (64 bit) and Suricata installed IDS mode.
> I used configuration as indicated in the document (
> https://redmine.openinfosecfoundation.org/projects/suricata/wiki/File_Extraction
> ).
>
> files.rules enabled in suricata.yaml the other rules disabled and file-store
> is enabled.
> In file.rules, enabled rules about exe file.
> request-body-limit and response-body-limit set to 0.
>
> I run the Suricata and I downloaded many executables but no file extracted.
> One example link : Download Pdf Reader (
> http://www.gezginler.net/indir/sumatra-pdf.html ) if needed I may add the
> pcap file.
> I checked all config and I tried over and over, watching fast.log, my rule
> has never matched. I couldn't find the problem.
> I should be missing a flag , please help me about how to find what config is
> missing.
>
> Sended some needed logs, part of config files and my rules at below.
> Thanks.
>
> part of suricata.yaml
>
> - file-store:
> enabled: yes # set to yes to enable
> log-dir: /home/coker/SuriOutDir # directory to store the files
> force-magic: no # force logging magic on all stored files
> force-md5: yes # force logging of md5 checksums
> waldo: file.waldo # waldo file to store the file_id across runs
>
> # output module to log files tracked in a easily parsable json format
> - file-log:
> enabled: yes
> filename: files-json.log
> append: yes
> #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'
>
> force-magic: no # force logging magic on all logged files
> force-md5: no # force logging of md5 checksums
> .
> .
> .
> rule-files:
> - myrules.rules
> # - botcc.rules
> # - ciarmy.rules
> ...
> myrules.rules
>
> alert http any any -> any any (msg:"Pattern based";
> flow:established,to_client; filemagic:"executable for MS Windows";
> filestore; sid:18; rev:1;)
> alert http any any -> any any (msg:"Extension based !";
> flow:established,to_client; fileext:"exe"; filemagic:"executable for MS
> Windows"; filestore; sid:19; rev:1;)
>
>
> suricata starting logs
>
> This is Suricata version 1.4.7 RELEASE
> 17/3/2014 -- 15:13:16 - <Info> - CPUs/cores online: 4
> 17/3/2014 -- 15:13:16 - <Info> - Found an MTU of 1500 for 'eth0'
> 17/3/2014 -- 15:13:16 - <Info> - allocated 3670016 bytes of memory for the
> defrag hash... 65536 buckets of size 56
> 17/3/2014 -- 15:13:16 - <Info> - preallocated 65535 defrag trackers of size
> 144
> 17/3/2014 -- 15:13:16 - <Info> - defrag memory usage: 13107056 bytes,
> maximum: 33554432
> 17/3/2014 -- 15:13:16 - <Info> - AutoFP mode using default "Active Packets"
> flow load balancer
> 17/3/2014 -- 15:13:16 - <Info> - preallocated 1024 packets. Total memory
> 4362240
> 17/3/2014 -- 15:13:16 - <Info> - allocated 229376 bytes of memory for the
> host hash... 4096 buckets of size 56
> 17/3/2014 -- 15:13:16 - <Info> - preallocated 1000 hosts of size 120
> 17/3/2014 -- 15:13:16 - <Info> - host memory usage: 349376 bytes, maximum:
> 16777216
> 17/3/2014 -- 15:13:16 - <Info> - allocated 3670016 bytes of memory for the
> flow hash... 65536 buckets of size 56
> 17/3/2014 -- 15:13:16 - <Info> - preallocated 10000 flows of size 272
> 17/3/2014 -- 15:13:16 - <Info> - flow memory usage: 6390016 bytes, maximum:
> 33554432
> 17/3/2014 -- 15:13:16 - <Info> - IP reputation disabled
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - Delayed detect disabled
> 17/3/2014 -- 15:13:16 - <Info> - 1 rule files processed. 2 rules
> successfully loaded, 0 rules failed
> 17/3/2014 -- 15:13:16 - <Info> - 2 signatures processed. 0 are IP-only
> rules, 0 are inspecting packet payload, 2 inspect application layer, 0 are
> decoder event only
> 17/3/2014 -- 15:13:16 - <Info> - building signature grouping structure,
> stage 1: adding signatures to signature source addresses... complete
> 17/3/2014 -- 15:13:16 - <Info> - building signature grouping structure,
> stage 2: building source address list... complete
> 17/3/2014 -- 15:13:16 - <Info> - building signature grouping structure,
> stage 3: building destination address lists... complete
> 17/3/2014 -- 15:13:16 - <Info> - Threshold config parsed: 0 rule(s) found
> 17/3/2014 -- 15:13:16 - <Info> - Core dump size set to unlimited.
> 17/3/2014 -- 15:13:16 - <Info> - fast output device (regular) initialized:
> fast.log
> 17/3/2014 -- 15:13:16 - <Info> - Unified2-alert initialized: filename
> unified2.alert, limit 32 MB
> 17/3/2014 -- 15:13:16 - <Info> - http-log output device (regular)
> initialized: http.log
> 17/3/2014 -- 15:13:16 - <Info> - md5 calculation requires linking against
> libnss
> 17/3/2014 -- 15:13:16 - <Info> - loading waldo file
> /var/log/suricata//file.waldo
> 17/3/2014 -- 15:13:16 - <Info> - id 2833
> 17/3/2014 -- 15:13:16 - <Info> - storing files in /home/coker/SuriOutDir
> 17/3/2014 -- 15:13:16 - <Info> - file-log output device (regular)
> initialized: files-json.log
> 17/3/2014 -- 15:13:16 - <Info> - Using 1 live device(s).
> 17/3/2014 -- 15:13:16 - <Info> - using interface eth0
> 17/3/2014 -- 15:13:16 - <Info> - Running in 'auto' checksum mode. Detection
> of interface state will require 1000 packets.
> 17/3/2014 -- 15:13:16 - <Info> - Found an MTU of 1500 for 'eth0'
> 17/3/2014 -- 15:13:16 - <Info> - Set snaplen to 1500 for 'eth0'
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - returning 0x7fbdf4004830
> 17/3/2014 -- 15:13:16 - <Info> - Created file drop directory
> /home/coker/SuriOutDir
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - returning 0x7fbdec0047c0
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - returning 0x7fbdf00047c0
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - returning 0x7fbde40047c0
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - returning 0x7fbde80047c0
> 17/3/2014 -- 15:13:16 - <Info> - using magic-file /usr/share/file/magic
> 17/3/2014 -- 15:13:16 - <Info> - returning 0x7fbddc0047c0
> 17/3/2014 -- 15:13:16 - <Info> - RunModeIdsPcapAutoFp initialised
> 17/3/2014 -- 15:13:16 - <Info> - stream "max-sessions": 262144
> 17/3/2014 -- 15:13:16 - <Info> - stream "prealloc-sessions": 32768
> 17/3/2014 -- 15:13:16 - <Info> - stream "memcap": 33554432
> 17/3/2014 -- 15:13:16 - <Info> - stream "midstream" session pickups:
> disabled
> 17/3/2014 -- 15:13:16 - <Info> - stream "async-oneside": disabled
> 17/3/2014 -- 15:13:16 - <Info> - stream "checksum-validation": enabled
> 17/3/2014 -- 15:13:16 - <Info> - stream."inline": disabled
> 17/3/2014 -- 15:13:16 - <Info> - stream.reassembly "memcap": 67108864
> 17/3/2014 -- 15:13:16 - <Info> - stream.reassembly "depth": 1048576
> 17/3/2014 -- 15:13:16 - <Info> - stream.reassembly "toserver-chunk-size":
> 2560
> 17/3/2014 -- 15:13:16 - <Info> - stream.reassembly "toclient-chunk-size":
> 2560
> 17/3/2014 -- 15:13:16 - <Info> - all 7 packet processing threads, 3
> management threads initialized, engine started.
> 17/3/2014 -- 15:13:16 - <Info> - No packets with invalid checksum, assuming
> checksum offloading is NOT used
>
> one line of files-json.log
>
> "timestamp": "03\/17\/2014-11:12:23.726393", "ipver": 4, "srcip":
> "213.180.204.224", "dstip": "10.41.0.196", "protocol": 6, "sp": 80, "dp":
> 53249, "http_uri":
> "\/clck\/click\/dtype=stred\/pid=1\/cid=72202\/reqid=20927.8233.1395047496.86614\/path=690.1033\/vars=143=1035.15.899,287=11508,1036=0,1037=0,1038=0,1039=550,1040=308,1041=574,1042=Mozilla\/5.0
> (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Ubuntu
> Chromium\/32.0.1700.107 Chrome\/32.0.1700.107
> Safari\/537.36,1051=1136,1040.318=901,1041.660=1167,1050.660=1518,1041.906=1167,1041.318=1167\/slots=\/*",
> "http_host": "yandex.com.tr", "http_referer":
> "http:\/\/www.yandex.com.tr\/", "http_user_agent": "Mozilla\/5.0 (X11; Linux
> x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Ubuntu
> Chromium\/32.0.1700.107 Chrome\/32.0.1700.107 Safari\/537.36", "filename":
> "\/clck\/click\/dtype=stred\/pid=1\/cid=72202\/reqid=20927.8233.1395047496.86614\/path=690.1033\/vars=143=1035.15.899,287=11508,1036=0,1037=0,1038=0,1039=550,1040=308,1041=574,1042=Mozilla\/5.0
> (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Ubuntu
> Chromium\/32.0.1700.107 Chrome\/32.0.1700.107
> Safari\/537.36,1051=1136,1040.318=901,1041.660=1167,1050.660=1518,1041.906=1167,1041.318=1167\/slots=\/*",
> "magic": "unknown", "state": "CLOSED", "stored": false, "size": 43
>
> some lines of http.log
> 03/17/2014-14:27:01.220478 www.gezginler.net [**]
> /oyunlar/resimler/indir/skill.jpg [**] Mozilla/5.0 (X11; Ubuntu; Linux
> x86_64; rv:27.0) Gecko/20100101 Firefox/27.0 [**] 10.41.0.196:56421 ->
> 50.22.202.134:80
> 03/17/2014-14:27:01.221176 www.gezginler.net [**]
> /tema/eklenti/qtip2/jquery.qtip.min.js [**] Mozilla/5.0 (X11; Ubuntu; Linux
> x86_64; rv:27.0) Gecko/20100101 Firefox/27.0 [**] 10.41.0.196:56295 ->
> 50.22.202.134:80
> 03/17/2014-14:27:01.222314 qa.sockets.stackexchange.com [**] / [**]
> Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko)
> Ubuntu Chromium/32.0.1700.107 Chrome/32.0.1700.107 Safari/537.36 [**]
> 10.41.0.196:54075 -> 198.252.206.25:80
> 03/17/2014-14:27:01.222314 <hostname unknown> [**] [**] <useragent unknown>
> [**] 10.41.0.196:54075 -> 198.252.206.25:80
> 03/17/2014-14:27:01.223333 www.gezginler.net [**]
> /oyunlar/resimler/indir/warthunder5.jpg [**] Mozilla/5.0 (X11; Linux x86_64)
> AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/32.0.1700.107
> Chrome/32.0.1700.107 Safari/537.36 [**] 10.41.0.196:56206 ->
> 50.22.202.134:80
> 03/17/2014-14:27:01.224830 www.gezginler.net [**] /tema/stil.css [**]
> Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0
> [**] 10.41.0.196:56294 -> 50.22.202.134:80
> 03/17/2014-14:27:01.225208 www.gezginler.net [**]
> /tema/eklenti/rating/jquery/jRating.jquery2.js [**] Mozilla/5.0 (X11;
> Ubuntu; Linux x86_64; rv:27.0) Gecko/20100101 Firefox/27.0 [**]
> 10.41.0.196:56376 -> 50.22.202.134:80
>
Do you have interface offloading disabled (ethtool -k eth0)?
Try it also with checksum validation - disabled (false)
thanks
--
Regards,
Peter Manev
More information about the Oisf-users
mailing list