[Oisf-users] File extraction problems (false positives)

[Joakim Kunst Forsbakk]

Hi,

I tried disabling all filestore rules, and tested the rule you suggested over one hour.
The fast log shows that the rule triggered 256 times in one hour.
Suricata however stored 1021 files. 248 of these are actual PDF files, but all the other files are ASCII text files, PNG image data, GIF image data, UTF-8 unicode text and XML-files.

Any idea why Suricata does this?

Best regards
Joakim Kunst Forsbakk


> -----Original Message-----
> From: Peter Manev [mailto:petermanev at gmail.com]
> Sent: 21. februar 2014 14:21
> To: Joakim Kunst Forsbakk
> Cc: oisf-users at lists.openinfosecfoundation.org
> Subject: Re: [Oisf-users] File extraction problems (false positives)
> 
> On Fri, Feb 21, 2014 at 2:03 PM, Joakim Kunst Forsbakk
> <forsbakk at mnemonic.no> wrote:
> > Hello all,
> >
> > I've been trying to get file extraction to work on some Suricata sensors for a
> while, and I am having some problems I haven't read about in the forum
> before.
> > I have rules to detect and store exe, zip and pdf files based on filemagic.
> Many of the downloaded files are correctly identified and stored, but most
> of the stored files are flase positives. The stored files are mostly png, jpeg,
> gif, html, ascii files and so on.
> >
> > Some example rules I use:
> > alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PDF
> > document"; flow:established,to_client; filemagic:"PDF document";
> > filestore:to_client,file; classtype:low-severity; sid:1240007; rev:2;)
> > alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PE32
> > executable"; flow:established,to_client; filemagic:"PE32 executable";
> > filestore:to_client,file; classtype:low-severity; sid:1200443; rev:
> > 1;) alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PE
> > for MS Windows"; flow:established,to_client; filemagic:"PE for MS
> > Windows"; filestore:to_client,file; classtype:low-severity;
> > sid:1220012; rev:2;) alert http $EXTERNAL_NET any -> $HOME_NET any
> > (msg:"FILEMAGIC Zip archive data"; flow:established,to_client;
> > filemagic:"Zip archive data"; filestore:to_client,file;
> > classtype:low-severity; sid:1230003; rev:2;)
> >
> 
> I apologize , my previous mail was sent too fast.
> Could you try that and see if any difference:
> alert http $EXTERNAL_NET any -> $HOME_NET any (msg:"FILEMAGIC PDF
> document"; filemagic:"PDF"; filestore; classtype:low-severity; sid:1240007;
> rev:2;)
> 
> 
> > I've also tested these, without seeing any big difference:
> > alert tcp $EXTERNAL_NET any -> $HOME_NET any (msg:"FILECARVING
> PDF";
> > flow:to_client,established; file_data; content:"%PDF-"; nocase;
> > filestore; classtype:low-severity; sid:1250001; rev:1;) alert tcp
> > $EXTERNAL_NET any -> $HOME_NET any (msg:" FILECARVING EXE";
> > flow:to_client,established; file_data; content:"MZ";
> > byte_jump:4,58,relative,little; content:"PE|00 00|"; within:4;
> > distance:-64; filestore; classtype:low-severity; sid:1250002; rev:1;)
> >
> > Any ideas on how to improve this and get rid of the false positives?
> >
> > I've tested on sensors in different networks. Most of the sensors are
> Suricata 1.4.6, but I've also tested 1.4.7. They have 32 cores, 64gb ram, Linux
> 3.12.0-1.el6.elrepo.x86_64 CentOS.
> >
> > Excerpt from suricata.yaml:
> >
> > af-packet:
> >   - interface: eth4
> >     threads: 16
> >     cluster-id: 94
> >     cluster-type: cluster_flow
> >     defrag: yes
> >     use-mmap: yes
> >     ring-size: 30000
> >     buffer-size: 128536
> >   - interface: eth5
> >     threads: 16
> >     cluster-id: 95
> >     cluster-type: cluster_flow
> >     defrag: yes
> >     use-mmap: yes
> >     ring-size: 30000
> >     buffer-size: 128536
> >
> > stream:
> >   memcap: 12gb
> >   checksum-validation: yes
> >   inline: auto
> >   reassembly:
> >     memcap: 12gb
> >     depth: 0
> >     toserver-chunk-size: 2560
> >     toclient-chunk-size: 2560
> >
> > libhtp:
> >    default-config:
> >      personality: IDS
> >      request-body-limit: 0
> >      response-body-limit: 0
> >
> > Thanks for any help.
> >
> > Best regards
> > Joakim
> > _______________________________________________
> > Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> > Site: http://suricata-ids.org | Support:
> > http://suricata-ids.org/support/
> > List:
> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> > OISF: http://www.openinfosecfoundation.org/
> 
> 
> 
> --
> Regards,
> Peter Manev