[Oisf-devel] Feature request: stream logging mode
mcholste at gmail.com
Mon Nov 28 14:01:30 UTC 2011
> This name implies the stream is not logged until it is complete,
> implying buffering. How does this work with large streams? Or is there
> some temp file that is renamed to the final filename when the stream ends.
That is correct--it does not log anything until the stream is cutoff
either by size (we use 1 MB) or by time (we use 60 seconds). I would
think you would re-purpose existing memcap settings to achieve this
effect. There are no temp files used.
> Most of the functionality exists already. One challenge I see is that
> pcap dumping using libpcap is not thread safe, so we'd either have to
> lock it (like we do in the log-pcap module now) or we'd have to write
> our own pcap dumping code. Shouldn't be that much work, the format is
> quite simple. It adds up to development time though.
Vortex does not log pcap, it logs raw streams (like you would see in
"Follow stream" output from Wireshark). Suricata only allows a single
thread to work on any given stream at a time, right? My thought was
that you could use something akin to --runmode=workers for this.
> Another doubt I have is about how to write to disk. We're talking lots
> of files here, potentially many many thousands simultaneously. Keeping
> file descriptors open might not be feasible at that point, but
> opening/closing files on a per packet basis is probably a performance
> hog. So we'd maybe have to buffer in memory so we (hopefully) can write
> out multiple packets at the same time.
This is where the stream vs. packets part is really differentiated.
We're talking streams, so only a few thousand per second on a busy
network, and their file descriptors are not opened until Suricata is
ready to write the full stream to disk, so FD's are not usually a
problem, as the stream is written and the file is closed right away.
With the stream cutoff, memory is not a problem either. Is that
making more sense now?
More information about the Oisf-devel