[Oisf-users] File Extraction issues

[Jeremy A. Grove]

Hi All, 

I have not been able to get file extraction working consistently. I run Suricata 3.2.2 in Debian. I have some of the more important settings below. The issue is that Suricata is able to see the PDF as evidenced in the log line below. I have the rule enabled and I have supplied the rule below. Suricata also recognizes that this file is a PDF as referenced in the "magic" section of the log. 

If I turn on other file types they seem to work. 

As a secondary issue, the few PDFs that have been extracted successfully have been truncated and seem incomplete. 

{"timestamp":"2017-07-19T18:27:46.365317+0000","flow_id":604394854682393,"event_type":"fileinfo","src_ip":"75.119.201.252","src_port":80,"dest_ip":"X.X.X.X","dest_port":43012,"proto":"TCP","http":{"hostname":"css4.pub","url":"\/2015\/icelandic\/dictionary.pdf","http_user_agent":"Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/59.0.3071.115 Safari\/537.36","http_content_type":"application\/pdf","http_refer":"http:\/\/www.princexml.com\/samples\/","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":162948},"app_proto":"http","fileinfo":{"filename":"\/2015\/icelandic\/dictionary.pdf","magic":"PDF document, version 1.4","state":"TRUNCATED","stored":false,"size":162948,"tx_id":1}} 

alert http any any -> any any (msg:"FILEMAGIC pdf"; flow:established,to_server; filemagic:"PDF document"; filestore; sid:9; rev:1;) 


- file-store: 
enabled: yes # set to yes to enable 
log-dir: files # directory to store the files 
force-magic: yes # force logging magic on all stored files 
force-md5: yes # force logging of md5 checksums 
force-filestore: no # force storing of all files 
#waldo: file.waldo # waldo file to store the file_id across runs 

- file-log: 
enabled: yes 
filename: files-json.log 
append: yes 
#filetype: regular # 'regular', 'unix_stream' or 'unix_dgram' 

force-magic: yes # force logging magic on all logged files 
force-md5: yes # force logging of md5 checksums 

libhtp: 
default-config: 
personality: IDS 

# Can be specified in kb, mb, gb. Just a number indicates 
# it's in bytes. 
request-body-limit: 0 
response-body-limit: 0 

stream: 
memcap: 64mb 
checksum-validation: no # reject wrong csums 
inline: auto # auto will use inline mode in IPS mode, yes or no set it statically 
reassembly: 
memcap: 256mb 
depth: 0 # reassemble 1mb into a stream 
toserver-chunk-size: 2560 
toclient-chunk-size: 2560 
randomize-chunk-size: yes 

Any advice on this? 

Regards, 

Jeremy Grove, SSCP 
Senior Information Security Analyst 
Quadrant Information Security 
o: [ callto:(904)296-9100 | (904)296-9100 ] x100 
t: [ callto:(800) 538-9357 | (800) 538-9357 ] x100 
e: [ mailto:soc at quadrantsec.com | soc at quadrantsec.com ] 

Learn more= about our managed SIEM [ https://a.quadrantsec.com/3D%22https://quadrantsec.com/SaganMSSP%22 | people + product ] 


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20170719/f6a283c3/attachment.html>