<html><body><div style="font-family: arial, helvetica, sans-serif; font-size: 12pt; color: #000000"><div>Hi All,</div><div><br data-mce-bogus="1"></div><div>I have not been able to get file extraction working consistently. I run Suricata 3.2.2 in Debian. I have some of the more important settings below. The issue is that Suricata is able to see the PDF as evidenced in the log line below. I have the rule enabled and I have supplied the rule below. Suricata also recognizes that this file is a PDF as referenced in the "magic" section of the log. </div><div><br data-mce-bogus="1"></div><div>If I turn on other file types they seem to work.</div><div><br data-mce-bogus="1"></div><div>As a secondary issue, the few PDFs that have been extracted successfully have been truncated and seem incomplete. </div><div><br data-mce-bogus="1"></div><div><div>{"timestamp":"2017-07-19T18:27:46.365317+0000","flow_id":604394854682393,"event_type":"fileinfo","src_ip":"75.119.201.252","src_port":80,"dest_ip":"X.X.X.X","dest_port":43012,"proto":"TCP","http":{"hostname":"css4.pub","url":"\/2015\/icelandic\/dictionary.pdf","http_user_agent":"Mozilla\/5.0 (X11; Linux x86_64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/59.0.3071.115 Safari\/537.36","http_content_type":"application\/pdf","http_refer":"http:\/\/www.princexml.com\/samples\/","http_method":"GET","protocol":"HTTP\/1.1","status":200,"length":162948},"app_proto":"http","fileinfo":{"filename":"\/2015\/icelandic\/dictionary.pdf","magic":"PDF document, version 1.4","state":"TRUNCATED","stored":false,"size":162948,"tx_id":1}}</div></div><div><br></div><div>alert http any any -> any any (msg:"FILEMAGIC pdf"; flow:established,to_server; filemagic:"PDF document"; filestore; sid:9; rev:1;)<br></div><div><br></div><div><br data-mce-bogus="1"></div><div><div> - file-store:</div><div> enabled: yes # set to yes to enable</div><div> log-dir: files # directory to store the files</div><div> force-magic: yes # force logging magic on all stored files</div><div> force-md5: yes # force logging of md5 checksums</div><div> force-filestore: no # force storing of all files</div><div> #waldo: file.waldo # waldo file to store the file_id across runs</div><div><br></div><div> - file-log:</div><div> enabled: yes</div><div> filename: files-json.log</div><div> append: yes</div><div> #filetype: regular # 'regular', 'unix_stream' or 'unix_dgram'</div><div><br></div><div> force-magic: yes # force logging magic on all logged files</div><div> force-md5: yes # force logging of md5 checksums</div><div><br></div><div><div> libhtp:</div><div> default-config:</div><div> personality: IDS</div><div><br></div><div> # Can be specified in kb, mb, gb. Just a number indicates</div><div> # it's in bytes.</div><div> request-body-limit: 0</div><div> response-body-limit: 0</div></div></div><div><br></div><div><div>stream:</div><div> memcap: 64mb</div><div> checksum-validation: no # reject wrong csums</div><div> inline: auto # auto will use inline mode in IPS mode, yes or no set it statically</div><div> reassembly:</div><div> memcap: 256mb</div><div> depth: 0 # reassemble 1mb into a stream</div><div> toserver-chunk-size: 2560</div><div> toclient-chunk-size: 2560</div><div> randomize-chunk-size: yes</div></div><div><br></div><div>Any advice on this?</div><div><br data-mce-bogus="1"></div><div>Regards,</div><div><br></div><div data-marker="__SIG_PRE__"><div><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Jeremy Grove, SSCP</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Senior Information Security Analyst</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Quadrant Information Security</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">o: </span><span class="Object" id="OBJ_PREFIX_DWT146_com_zimbra_phone" style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><a href="callto:(904)296-9100" style="color: #005a95; text-decoration: none; cursor: pointer;" target="_blank" data-mce-style="color: #005a95; text-decoration: none; cursor: pointer;">(904)296-9100</a></span><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"> x100</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">t: </span><span class="Object" id="OBJ_PREFIX_DWT147_com_zimbra_phone" style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><a href="callto:(800) 538-9357" style="color: #005a95; text-decoration: none; cursor: pointer;" target="_blank" data-mce-style="color: #005a95; text-decoration: none; cursor: pointer;">(800) 538-9357</a></span><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"> x100</span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">e:</span><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"> </span><span class="Object" id="OBJ_PREFIX_DWT148_ZmEmailObjectHandler" style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="color: #005a95; cursor: pointer; font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><a class="moz-txt-link-abbreviated" href="mailto:soc@quadrantsec.com" target="_blank" style="color: #005a95; text-decoration: none; cursor: pointer;" data-mce-style="color: #005a95; text-decoration: none; cursor: pointer;">soc@quadrantsec.com</a></span><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><br style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;"><span style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;" data-mce-style="font-family: 'Segoe UI', 'Lucida Sans', sans-serif; font-size: 14.16px;">Learn more= about our managed SIEM <span class="Object" id="OBJ_PREFIX_DWT149_com_zimbra_url" style="color: #005a95; cursor: pointer;" data-mce-style="color: #005a95; cursor: pointer;"><a href="https://a.quadrantsec.com/3D%22https://quadrantsec.com/SaganMSSP%22" target="_blank" style="color: #005a95; text-decoration: none; cursor: pointer;" data-mce-style="color: #005a95; text-decoration: none; cursor: pointer;">people + product</a></span></span><br><br><br></div></div></div></body></html>