[Oisf-users] Suricata 1.4 Meta Files and Data Files produced from File Extraction

[Vincent Fang]

Ok that's good to know. I found one oddity with the file extraction with
magic and md5 hash enabled in the Suricata.yaml file and I'm not sure if
it's a bug or an issue on my end.

The alert signature looks like this

alert ip any any -> any any (msg:"file store all"; filestore; sid:5;)

and I examined all the meta and data files in the files directory
/usr/local/var/log/suricata/files

and I ran the linux command after observing all meta data files had 16 lines

wc -l *.meta | grep -v 16

and I came across one meta file that only had 12 lines that was missing the
last 4 lines MAGIC, STATE, MD5, and SIZE. The pcap file I was running
Suricata against in offline mode was me visiting a businessweek.com site.
Does anyone know why this oddity would have occurred?

Vince


On Fri, Jun 7, 2013 at 4:05 AM, Victor Julien <lists at inliniac.net> wrote:

> On 06/07/2013 09:41 AM, Vincent Fang wrote:
> > We're trying to use the meta files and data files created by Suricata to
> > send data to one of our servers. However, we're running into an issue
> > where if we open a file too early that we get incomplete data either
> > from the data file or meta file. Note that we also have force-magic, MD5
> > hash, and file extraction as the enabled states in our Suricata.yaml
> file.
> >
> > What condition can we assume to be true so that we can open and read the
> > meta file and the data file safely without it being incomplete?
> >
> > Using python as our scripting language to access those files, I assumed
> > that if the data file was done, that all the data in the meta file would
> > be complete as well, but I get scenarios where the MAGIC, STATE, MD5,
> > and SIZE were missing. I'm assuming this is because Suricata is
> > calculating those values from the data file, then reopening the meta
> > file and adding those last values in?
>
> I would recommend using the json output. It contains the same info, but
> is written in one go.
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20130610/16d7f3ce/attachment-0002.html>