[Oisf-users] File extraction problems (false positives)

[Joakim Kunst Forsbakk]

Hello all,

I've been trying to get file extraction to work on some Suricata sensors for a while, and I am having some problems I haven't read about in the forum before.
I have rules to detect and store exe, zip and pdf files based on filemagic. Many of the downloaded files are correctly identified and stored, but most of the stored files are flase positives. The stored files are mostly png, jpeg, gif, html, ascii files and so on.

Some example rules I use:
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PDF document"; flow:established,to_client; filemagic:"PDF document"; filestore:to_client,file; classtype:low-severity; sid:1240007; rev:2;)
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PE32 executable"; flow:established,to_client; filemagic:"PE32 executable"; filestore:to_client,file; classtype:low-severity; sid:1200443; rev: 1;)
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PE for MS Windows"; flow:established,to_client; filemagic:"PE for MS Windows"; filestore:to_client,file; classtype:low-severity; sid:1220012; rev:2;)
alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC Zip archive data"; flow:established,to_client; filemagic:"Zip archive data"; filestore:to_client,file; classtype:low-severity; sid:1230003; rev:2;)

I've also tested these, without seeing any big difference:
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"FILECARVING PDF"; flow:to_client,established; file_data; content:"%PDF-"; nocase; filestore; classtype:low-severity; sid:1250001; rev:1;)
alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:" FILECARVING EXE"; flow:to_client,established; file_data; content:"MZ"; byte_jump:4,58,relative,little; content:"PE|00 00|"; within:4; distance:-64; filestore; classtype:low-severity; sid:1250002; rev:1;)

Any ideas on how to improve this and get rid of the false positives?

I've tested on sensors in different networks. Most of the sensors are Suricata 1.4.6, but I've also tested 1.4.7. They have 32 cores, 64gb ram, Linux 3.12.0-1.el6.elrepo.x86_64 CentOS.

Excerpt from suricata.yaml:

af-packet:
  - interface: eth4
    threads: 16
    cluster-id: 94
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    ring-size: 30000
    buffer-size: 128536
  - interface: eth5
    threads: 16
    cluster-id: 95
    cluster-type: cluster_flow
    defrag: yes
    use-mmap: yes
    ring-size: 30000
    buffer-size: 128536

stream:
  memcap: 12gb
  checksum-validation: yes
  inline: auto
  reassembly:
    memcap: 12gb
    depth: 0 
    toserver-chunk-size: 2560
    toclient-chunk-size: 2560

libhtp:
   default-config:
     personality: IDS
     request-body-limit: 0
     response-body-limit: 0

Thanks for any help.

Best regards
Joakim