[Discussion] Cooperative event loops (e.g., libevent) to support asynch I/O

[Victor Julien]

I think the solution of having a time keeping thread that can execute
tasks scheduled by modules would actually lower the overhead. If we
would go for a SIGHUP solution, every part of the system would have to
ask itself "was this for me? Do I have to do something now?" Doing that
in a dedicated very low overhead thread that wakes up only on the times
something needs to be actually done seems more efficient to me.

In my view a HUP would be best for a general "reconfigure yourself", for
example rereading the config and rules files, etc...

Regards,
Victor

Martin Holste wrote:
> Ok, well other than the garbage collection, it sounds like all of those
> are good candidates for going in the "application layer" detached
> processes.  As long as the main detection process allows itself to be
> reconfigured on the fly, then there's no reason to bog it down with
> administrative tasks.  So, I think the question then becomes one of how
> do we best allow for a running configuration to be updated?  Certainly a
> SIGHUP would be the simplest, but do we need something less disruptive?
> 
> On Wed, Feb 11, 2009 at 3:10 AM, Victor Julien <lists at inliniac.net
> <mailto:lists at inliniac.net>> wrote:
> 
>     Martin Holste wrote:
>     > What kinds of tasks were you imagining would be executed at intervals?
>     > Garbage collection, statistics?  What did you have in mind?
> 
>     In Snort_inline I first ran into this when creating the clamav plugin
>     (not the best of ideas ;-)). I wanted it to check for updated virus defs
>     on an interval. Like Martin Fong wrote that was done using the packets
>     timestamp but the code was never activated if there were no packets. Not
>     a big problem in this case.
> 
>     Another example is a contract gig I did for a company, where they wanted
>     a preprocessor to report some stats to a http server on a fixed
>     interval, no matter if there had been packets or not.
> 
>     Garbage collection is a good point too. For example cleaning up expired
>     state... (flow/stream tables come to mind)...
> 
>     Regards,
>     Victor
> 
>     >
>     > On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net
>     <mailto:lists at inliniac.net>
>     > <mailto:lists at inliniac.net <mailto:lists at inliniac.net>>> wrote:
>     >
>     >     Martin Fong wrote:
>     >     > Matt Jonkman wrote:
>     >     >
>     >     >> Cooperative event loops (e.g., libevent) to support asynch I/O
>     >     >>
>     >     >> Do you mean something like parallel processing of different
>     tools
>     >     on the
>     >     >> same stream or event? For instance one stream with http
>     would go
>     >     through
>     >     >> regular matching but also be copied out to a http
>     interpreter to pull
>     >     >> environment variables and the like onto a separate
>     thread/processor?
>     >     >
>     >     > I'm actually thinking about unix select () loops.  Because
>     the current
>     >     > implementation simply invokes pcap_loop (), there's no way for a
>     >     > module to independently perform I/O or respond to a select
>     ()-based
>     >     > timer.  There have been cases when I've needed to write data
>     to a
>     >     > socket or read from a pipe, but my preprocessor only got the
>     >     > opportunity when it "processed" a packet (-- and clearly I
>     don't want
>     >     > the I/O to block).  (And, yes, I know about the danger of
>     having a
>     >     > module consume too many cycles doing I/O or computing, but
>     there's
>     >     > nothing to prevents such abuse now -- and cooperative event
>     processing
>     >     > _does_ work.)
>     >     >
>     >     >      As far as your suggestion, that too sounds interesting...!
>     >
>     >     I was thinking about having a way for module to register
>     functions that
>     >     will be executed at given times and intervals, probably by
>     having a
>     >     separate thread for that. That would introduce a lot of
>     locking issues
>     >     of course, so maybe another/additional way is needed.
>     >
>     >     What about having a module get called at a fixed interval even
>     if there
>     >     are no packets? Pcap/ip_queue/nf_queue etc can all work with
>     timeouts or
>     >     non-blocking io...
>     >
>     >     Maybe the module registration functions should be able to
>     determine the
>     >     interval, so that if no module requires it the overhead is not
>     imposed,
>     >     and if your module does it's set and used.
>     >
>     >     Interesting...
>     >
>     >     Cheers,
>     >     Victor
>     >
>     >     --
>     >     ---------------------------------------------
>     >     Victor Julien
>     >     http://www.inliniac.net/
>     >     PGP: http://www.inliniac.net/victorjulien.asc
>     >     ---------------------------------------------
>     >
>     >     _______________________________________________
>     >     Discussion mailing list
>     >     Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>
>     >     <mailto:Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>>
>     >     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>     >
>     >
> 
> 
>     --
>     ---------------------------------------------
>     Victor Julien
>     http://www.inliniac.net/
>     PGP: http://www.inliniac.net/victorjulien.asc
>     ---------------------------------------------
> 
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------