[Oisf-users] Suricata 1.4 Meta Files and Data Files produced from File Extraction

Peter Manev petermanev at gmail.com
Fri Jun 7 07:48:03 UTC 2013

On Fri, Jun 7, 2013 at 9:41 AM, Vincent Fang <vincent.y.fang at gmail.com> wrote:
> We're trying to use the meta files and data files created by Suricata to
> send data to one of our servers. However, we're running into an issue where
> if we open a file too early that we get incomplete data either from the data
> file or meta file. Note that we also have force-magic, MD5 hash, and file
> extraction as the enabled states in our Suricata.yaml file.
> What condition can we assume to be true so that we can open and read the
> meta file and the data file safely without it being incomplete?
> Using python as our scripting language to access those files, I assumed that
> if the data file was done, that all the data in the meta file would be
> complete as well, but I get scenarios where the MAGIC, STATE, MD5, and SIZE
> were missing. I'm assuming this is because Suricata is calculating those
> values from the data file, then reopening the meta file and adding those
> last values in?
> Vince
> _______________________________________________
> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> Site: http://suricata-ids.org | Support: http://suricata-ids.org/support/
> List: https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> OISF: http://www.openinfosecfoundation.org/

Hi ,

I am guessing the meta data is written in some sort of  "chunk"
fashion - where  a few metadata fields could be written  at the same
time  after some condition (buffer cleaning , mem limit ..so on) is

I could recommend (testing) to send the data (files and their
corresponding meta data) in the following pseudo manner (a wild
example) -
if file is older than 5 min - send it

This is just a quick and dirty approach off the top of my head.


Peter Manev

More information about the Oisf-users mailing list