[Discussion] Cooperative event loops (e.g., libevent) to support asynch I/O

[Victor Julien]

Martin Holste wrote:
> What kinds of tasks were you imagining would be executed at intervals? 
> Garbage collection, statistics?  What did you have in mind?

In Snort_inline I first ran into this when creating the clamav plugin
(not the best of ideas ;-)). I wanted it to check for updated virus defs
on an interval. Like Martin Fong wrote that was done using the packets
timestamp but the code was never activated if there were no packets. Not
a big problem in this case.

Another example is a contract gig I did for a company, where they wanted
a preprocessor to report some stats to a http server on a fixed
interval, no matter if there had been packets or not.

Garbage collection is a good point too. For example cleaning up expired
state... (flow/stream tables come to mind)...

Regards,
Victor

> 
> On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net
> <mailto:lists at inliniac.net>> wrote:
> 
>     Martin Fong wrote:
>     > Matt Jonkman wrote:
>     >
>     >> Cooperative event loops (e.g., libevent) to support asynch I/O
>     >>
>     >> Do you mean something like parallel processing of different tools
>     on the
>     >> same stream or event? For instance one stream with http would go
>     through
>     >> regular matching but also be copied out to a http interpreter to pull
>     >> environment variables and the like onto a separate thread/processor?
>     >
>     > I'm actually thinking about unix select () loops.  Because the current
>     > implementation simply invokes pcap_loop (), there's no way for a
>     > module to independently perform I/O or respond to a select ()-based
>     > timer.  There have been cases when I've needed to write data to a
>     > socket or read from a pipe, but my preprocessor only got the
>     > opportunity when it "processed" a packet (-- and clearly I don't want
>     > the I/O to block).  (And, yes, I know about the danger of having a
>     > module consume too many cycles doing I/O or computing, but there's
>     > nothing to prevents such abuse now -- and cooperative event processing
>     > _does_ work.)
>     >
>     >      As far as your suggestion, that too sounds interesting...!
> 
>     I was thinking about having a way for module to register functions that
>     will be executed at given times and intervals, probably by having a
>     separate thread for that. That would introduce a lot of locking issues
>     of course, so maybe another/additional way is needed.
> 
>     What about having a module get called at a fixed interval even if there
>     are no packets? Pcap/ip_queue/nf_queue etc can all work with timeouts or
>     non-blocking io...
> 
>     Maybe the module registration functions should be able to determine the
>     interval, so that if no module requires it the overhead is not imposed,
>     and if your module does it's set and used.
> 
>     Interesting...
> 
>     Cheers,
>     Victor
> 
>     --
>     ---------------------------------------------
>     Victor Julien
>     http://www.inliniac.net/
>     PGP: http://www.inliniac.net/victorjulien.asc
>     ---------------------------------------------
> 
>     _______________________________________________
>     Discussion mailing list
>     Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>
>     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> 
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------