[Oisf-devel] Feature request: stream logging mode

[Victor Julien]

On 11/28/2011 05:28 AM, Martin Holste wrote:
>> Can you explain in some detail what Suricata would need to output? I
>> read on the streamdb site it creates an indexed db and a data file.
>> Whats the format of each?
> 
> StreamDB takes care of all of the indexes, database work, and
> everything not involved with collecting the stream off of the network.
>  All I need Suricata to do is write the stream buffer to disk with a
> file named something like this (Vortex output):
> tcp-100243088-1321557507-1321557516-r-9940-10.0.145.126:52589s66.235.132.118:80
> $proto-$cnx_id-$start_ts-$end_ts-$term_reason-$bytes-$srcip:$srcport$originator$dstip:$dstport

This name implies the stream is not logged until it is complete,
implying buffering. How does this work with large streams? Or is there
some temp file that is renamed to the final filename when the stream ends.

> Then print (or log) the name of the file so that the wrapper script
> doesn't have to perform expensive directory listings to find the new
> files being created.  StreamDB reads the file as written by Suricata,
> and appends the data to the current stream file, then writes the file
> offset and connection information to the database.
> 
> I'm hoping that since Suricata has already organized the data into
> streams that this would be an easy win.

Most of the functionality exists already. One challenge I see is that
pcap dumping using libpcap is not thread safe, so we'd either have to
lock it (like we do in the log-pcap module now) or we'd have to write
our own pcap dumping code. Shouldn't be that much work, the format is
quite simple. It adds up to development time though.

Another doubt I have is about how to write to disk. We're talking lots
of files here, potentially many many thousands simultaneously. Keeping
file descriptors open might not be feasible at that point, but
opening/closing files on a per packet basis is probably a performance
hog. So we'd maybe have to buffer in memory so we (hopefully) can write
out multiple packets at the same time.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------