[Discussion] Cooperative event loops (e.g., libevent) to support asynch I/O

[Martin Holste]

Ok, well other than the garbage collection, it sounds like all of those are
good candidates for going in the "application layer" detached processes.  As
long as the main detection process allows itself to be reconfigured on the
fly, then there's no reason to bog it down with administrative tasks.  So, I
think the question then becomes one of how do we best allow for a running
configuration to be updated?  Certainly a SIGHUP would be the simplest, but
do we need something less disruptive?

On Wed, Feb 11, 2009 at 3:10 AM, Victor Julien <lists at inliniac.net> wrote:

> Martin Holste wrote:
> > What kinds of tasks were you imagining would be executed at intervals?
> > Garbage collection, statistics?  What did you have in mind?
>
> In Snort_inline I first ran into this when creating the clamav plugin
> (not the best of ideas ;-)). I wanted it to check for updated virus defs
> on an interval. Like Martin Fong wrote that was done using the packets
> timestamp but the code was never activated if there were no packets. Not
> a big problem in this case.
>
> Another example is a contract gig I did for a company, where they wanted
> a preprocessor to report some stats to a http server on a fixed
> interval, no matter if there had been packets or not.
>
> Garbage collection is a good point too. For example cleaning up expired
> state... (flow/stream tables come to mind)...
>
> Regards,
> Victor
>
> >
> > On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net
> > <mailto:lists at inliniac.net>> wrote:
> >
> >     Martin Fong wrote:
> >     > Matt Jonkman wrote:
> >     >
> >     >> Cooperative event loops (e.g., libevent) to support asynch I/O
> >     >>
> >     >> Do you mean something like parallel processing of different tools
> >     on the
> >     >> same stream or event? For instance one stream with http would go
> >     through
> >     >> regular matching but also be copied out to a http interpreter to
> pull
> >     >> environment variables and the like onto a separate
> thread/processor?
> >     >
> >     > I'm actually thinking about unix select () loops.  Because the
> current
> >     > implementation simply invokes pcap_loop (), there's no way for a
> >     > module to independently perform I/O or respond to a select ()-based
> >     > timer.  There have been cases when I've needed to write data to a
> >     > socket or read from a pipe, but my preprocessor only got the
> >     > opportunity when it "processed" a packet (-- and clearly I don't
> want
> >     > the I/O to block).  (And, yes, I know about the danger of having a
> >     > module consume too many cycles doing I/O or computing, but there's
> >     > nothing to prevents such abuse now -- and cooperative event
> processing
> >     > _does_ work.)
> >     >
> >     >      As far as your suggestion, that too sounds interesting...!
> >
> >     I was thinking about having a way for module to register functions
> that
> >     will be executed at given times and intervals, probably by having a
> >     separate thread for that. That would introduce a lot of locking
> issues
> >     of course, so maybe another/additional way is needed.
> >
> >     What about having a module get called at a fixed interval even if
> there
> >     are no packets? Pcap/ip_queue/nf_queue etc can all work with timeouts
> or
> >     non-blocking io...
> >
> >     Maybe the module registration functions should be able to determine
> the
> >     interval, so that if no module requires it the overhead is not
> imposed,
> >     and if your module does it's set and used.
> >
> >     Interesting...
> >
> >     Cheers,
> >     Victor
> >
> >     --
> >     ---------------------------------------------
> >     Victor Julien
> >     http://www.inliniac.net/
> >     PGP: http://www.inliniac.net/victorjulien.asc
> >     ---------------------------------------------
> >
> >     _______________________________________________
> >     Discussion mailing list
> >     Discussion at openinfosecfoundation.org
> >     <mailto:Discussion at openinfosecfoundation.org>
> >     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> >
> >
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090211/d3a830be/attachment-0002.html>