[Oisf-users] Suricata 1.4 Meta Files and Data Files produced from File Extraction

[Peter Manev]

On Fri, Jun 7, 2013 at 9:41 AM, Vincent Fang <vincent.y.fang at gmail.com> wrote:
> We're trying to use the meta files and data files created by Suricata to
> send data to one of our servers. However, we're running into an issue where
> if we open a file too early that we get incomplete data either from the data
> file or meta file. Note that we also have force-magic, MD5 hash, and file
> extraction as the enabled states in our Suricata.yaml file.
>
> What condition can we assume to be true so that we can open and read the
> meta file and the data file safely without it being incomplete?
>
> Using python as our scripting language to access those files, I assumed that
> if the data file was done, that all the data in the meta file would be
> complete as well, but I get scenarios where the MAGIC, STATE, MD5, and SIZE
> were missing. I'm assuming this is because Suricata is calculating those
> values from the data file, then reopening the meta file and adding those
> last values in?
>
>
> Vince
>
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/


Hi ,

I am guessing the meta data is written in some sort of  "chunk"
fashion - where  a few metadata fields could be written  at the same
time  after some condition (buffer cleaning , mem limit ..so on) is
met.

I could recommend (testing) to send the data (files and their
corresponding meta data) in the following pseudo manner (a wild
example) -
if file is older than 5 min - send it

This is just a quick and dirty approach off the top of my head.

thanks


--
Regards,
Peter Manev