[Oisf-devel] Feature request: stream logging mode

[Victor Julien]

On 11/28/2011 03:01 PM, Martin Holste wrote:
>> This name implies the stream is not logged until it is complete,
>> implying buffering. How does this work with large streams? Or is there
>> some temp file that is renamed to the final filename when the stream ends.
>>
> That is correct--it does not log anything until the stream is cutoff
> either by size (we use 1 MB) or by time (we use 60 seconds).  I would
> think you would re-purpose existing memcap settings to achieve this
> effect.  There are no temp files used.

Do you expect the raw data or the reassembled stream (which does away
with overlapping packets, retransmissions, etc).

Our stream data lives in a sliding window currently, but keeping it up
to 1MB or so shouldn't be hard to implement.

The timeout shouldn't be hard either.

>> Most of the functionality exists already. One challenge I see is that
>> pcap dumping using libpcap is not thread safe, so we'd either have to
>> lock it (like we do in the log-pcap module now) or we'd have to write
>> our own pcap dumping code. Shouldn't be that much work, the format is
>> quite simple. It adds up to development time though.
>>
> 
> Vortex does not log pcap, it logs raw streams (like you would see in
> "Follow stream" output from Wireshark).  Suricata only allows a single
> thread to work on any given stream at a time, right?  My thought was
> that you could use something akin to --runmode=workers for this.

Right. We have that data in memory in the stream engine.

An autofp or workers style runmode would likely be best for a feature
like this.

>> Another doubt I have is about how to write to disk. We're talking lots
>> of files here, potentially many many thousands simultaneously. Keeping
>> file descriptors open might not be feasible at that point, but
>> opening/closing files on a per packet basis is probably a performance
>> hog. So we'd maybe have to buffer in memory so we (hopefully) can write
>> out multiple packets at the same time.
> 
> This is where the stream vs. packets part is really differentiated.
> We're talking streams, so only a few thousand per second on a busy
> network, and their file descriptors are not opened until Suricata is
> ready to write the full stream to disk, so FD's are not usually a
> problem, as the stream is written and the file is closed right away.
> With the stream cutoff, memory is not a problem either.  Is that
> making more sense now?
> 

Yup.

Can you open a feature ticket for this? Sounds like a low to medium
resource task. Priority is TBD :)

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------