[Oisf-devel] Feature request: stream logging mode

[Martin Holste]

> This name implies the stream is not logged until it is complete,
> implying buffering. How does this work with large streams? Or is there
> some temp file that is renamed to the final filename when the stream ends.
>
That is correct--it does not log anything until the stream is cutoff
either by size (we use 1 MB) or by time (we use 60 seconds).  I would
think you would re-purpose existing memcap settings to achieve this
effect.  There are no temp files used.

> Most of the functionality exists already. One challenge I see is that
> pcap dumping using libpcap is not thread safe, so we'd either have to
> lock it (like we do in the log-pcap module now) or we'd have to write
> our own pcap dumping code. Shouldn't be that much work, the format is
> quite simple. It adds up to development time though.
>

Vortex does not log pcap, it logs raw streams (like you would see in
"Follow stream" output from Wireshark).  Suricata only allows a single
thread to work on any given stream at a time, right?  My thought was
that you could use something akin to --runmode=workers for this.

> Another doubt I have is about how to write to disk. We're talking lots
> of files here, potentially many many thousands simultaneously. Keeping
> file descriptors open might not be feasible at that point, but
> opening/closing files on a per packet basis is probably a performance
> hog. So we'd maybe have to buffer in memory so we (hopefully) can write
> out multiple packets at the same time.

This is where the stream vs. packets part is really differentiated.
We're talking streams, so only a few thousand per second on a busy
network, and their file descriptors are not opened until Suricata is
ready to write the full stream to disk, so FD's are not usually a
problem, as the stream is written and the file is closed right away.
With the stream cutoff, memory is not a problem either.  Is that
making more sense now?