From jonkman at jonkmans.com  Wed Feb  4 14:33:10 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Wed, 04 Feb 2009 14:33:10 -0500
Subject: [Discussion] Submitted Ideas
Message-ID: <4989ED76.7080703@jonkmans.com>

Martin Fong of SRI sent in a list of some very good ideas. I'll post
them below and lets discuss a bit. I'm sure Martin can add to it as we go.


    - Content-based alert message substitution
    - Non-combinatoric IP/port lists
    - Cooperative event loops (e.g., libevent) to support asynch I/O
    - On-the-fly rule updates without state loss
    - Configuration file conditional preprocessor
    - Variable blackboards
    - Non-tokenized preprocessor parameter lines

Thanks Martin!

Matt

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From mcholste at gmail.com  Wed Feb  4 16:19:43 2009
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 4 Feb 2009 15:19:43 -0600
Subject: [Discussion] Submitted Ideas
In-Reply-To: <4989ED76.7080703@jonkmans.com>
References: <4989ED76.7080703@jonkmans.com>
Message-ID: <df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>

Those last two need quite a bit of explanation (at least for me)...

On Wed, Feb 4, 2009 at 1:33 PM, Matt Jonkman <jonkman at jonkmans.com> wrote:

> Martin Fong of SRI sent in a list of some very good ideas. I'll post
> them below and lets discuss a bit. I'm sure Martin can add to it as we go.
>
>
>    - Content-based alert message substitution
>    - Non-combinatoric IP/port lists
>    - Cooperative event loops (e.g., libevent) to support asynch I/O
>    - On-the-fly rule updates without state loss
>    - Configuration file conditional preprocessor
>    - Variable blackboards
>    - Non-tokenized preprocessor parameter lines
>
> Thanks Martin!
>
> Matt
>
> --
> --------------------------------------------
> Matthew Jonkman
> Emerging Threats
> Phone 765-429-0398
> Fax 312-264-0205
> http://www.emergingthreats.net
> --------------------------------------------
>
> PGP: http://www.jonkmans.com/mattjonkman.asc
>
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090204/3d85bd51/attachment.html

From lists at inliniac.net  Thu Feb  5 01:25:10 2009
From: lists at inliniac.net (Victor Julien)
Date: Thu, 05 Feb 2009 07:25:10 +0100
Subject: [Discussion] Submitted Ideas
In-Reply-To: <df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>
References: <4989ED76.7080703@jonkmans.com>
	<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>
Message-ID: <498A8646.4020902@inliniac.net>

Same here, for all of them though, except the alert message substitution
and on the fly rule updates...

It all sounds very interesting... if I only knew what it meant ;-)

Regards,
Victor

Martin Holste wrote:
> Those last two need quite a bit of explanation (at least for me)...
> 
> On Wed, Feb 4, 2009 at 1:33 PM, Matt Jonkman <jonkman at jonkmans.com
> <mailto:jonkman at jonkmans.com>> wrote:
> 
>     Martin Fong of SRI sent in a list of some very good ideas. I'll post
>     them below and lets discuss a bit. I'm sure Martin can add to it as
>     we go.
> 
> 
>        - Content-based alert message substitution
>        - Non-combinatoric IP/port lists
>        - Cooperative event loops (e.g., libevent) to support asynch I/O
>        - On-the-fly rule updates without state loss
>        - Configuration file conditional preprocessor
>        - Variable blackboards
>        - Non-tokenized preprocessor parameter lines
> 
>     Thanks Martin!
> 
>     Matt
> 
>     --
>     --------------------------------------------
>     Matthew Jonkman
>     Emerging Threats
>     Phone 765-429-0398
>     Fax 312-264-0205
>     http://www.emergingthreats.net
>     --------------------------------------------
> 
>     PGP: http://www.jonkmans.com/mattjonkman.asc
> 
> 
>     _______________________________________________
>     Discussion mailing list
>     Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>
>     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From jonkman at jonkmans.com  Thu Feb  5 11:43:19 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:43:19 -0500
Subject: [Discussion] Submitted Ideas
In-Reply-To: <498A8646.4020902@inliniac.net>
References: <4989ED76.7080703@jonkmans.com>	<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>
	<498A8646.4020902@inliniac.net>
Message-ID: <498B1727.8000004@jonkmans.com>

I'll divide these up into some separate emails and we can discuss....

Victor Julien wrote:
> Same here, for all of them though, except the alert message substitution
> and on the fly rule updates...
> 
> It all sounds very interesting... if I only knew what it meant ;-)
> 
> Regards,
> Victor
> 
> Martin Holste wrote:
>> Those last two need quite a bit of explanation (at least for me)...
>>
>> On Wed, Feb 4, 2009 at 1:33 PM, Matt Jonkman <jonkman at jonkmans.com
>> <mailto:jonkman at jonkmans.com>> wrote:
>>
>>     Martin Fong of SRI sent in a list of some very good ideas. I'll post
>>     them below and lets discuss a bit. I'm sure Martin can add to it as
>>     we go.
>>
>>
>>        - Content-based alert message substitution
>>        - Non-combinatoric IP/port lists
>>        - Cooperative event loops (e.g., libevent) to support asynch I/O
>>        - On-the-fly rule updates without state loss
>>        - Configuration file conditional preprocessor
>>        - Variable blackboards
>>        - Non-tokenized preprocessor parameter lines
>>
>>     Thanks Martin!
>>
>>     Matt
>>
>>     --
>>     --------------------------------------------
>>     Matthew Jonkman
>>     Emerging Threats
>>     Phone 765-429-0398
>>     Fax 312-264-0205
>>     http://www.emergingthreats.net
>>     --------------------------------------------
>>
>>     PGP: http://www.jonkmans.com/mattjonkman.asc
>>
>>
>>     _______________________________________________
>>     Discussion mailing list
>>     Discussion at openinfosecfoundation.org
>>     <mailto:Discussion at openinfosecfoundation.org>
>>     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>>
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Discussion mailing list
>> Discussion at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> 
> 

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:45:08 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:45:08 -0500
Subject: [Discussion] Content-based alert message substitution
Message-ID: <498B1794.7060203@jonkmans.com>

First of Martin's ideas:

Content-based alert message substitution


I like this a lot. Being able to pull a username out of the packet and
put it in the alert. Of course this leaves a lot of opportunities for
injection attacks against the event manager, but that can be handled if
we're careful.



-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:45:49 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:45:49 -0500
Subject: [Discussion] Non-combinatoric IP/port lists
Message-ID: <498B17BD.1080508@jonkmans.com>

Martin, can you elaborate on this one? Not sure what you're getting at.

Non-combinatoric IP/port lists

Thanks!

Matt
-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:47:21 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:47:21 -0500
Subject: [Discussion] Cooperative event loops (e.g.,
	libevent) to support asynch I/O
Message-ID: <498B1819.9010304@jonkmans.com>

Cooperative event loops (e.g., libevent) to support asynch I/O

Do you mean something like parallel processing of different tools on the
same stream or event? For instance one stream with http would go through
regular matching but also be copied out to a http interpreter to pull
environment variables and the like onto a separate thread/processor?

Matt


-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:48:04 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:48:04 -0500
Subject: [Discussion] On-the-fly rule updates without state loss
Message-ID: <498B1844.4050607@jonkmans.com>

On-the-fly rule updates without state loss

This is one I think we've discussed. Definitely a goal to shoot for.
What kind of challenges will we face placing a new ruleset into an
existing state?

Matt

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:48:42 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:48:42 -0500
Subject: [Discussion] Configuration file conditional preprocessor
Message-ID: <498B186A.7000005@jonkmans.com>

Configuration file conditional preprocessor

Do you mean a preprocessor that'd sanity check or modify the configuration?

Matt

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:49:12 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:49:12 -0500
Subject: [Discussion] Variable blackboards
Message-ID: <498B1888.1030606@jonkmans.com>

Variable blackboards

I assume you mean being able to pull real data into variables and share
those among streams?

Matt


-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From jonkman at jonkmans.com  Thu Feb  5 11:49:39 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 11:49:39 -0500
Subject: [Discussion] Non-tokenized preprocessor parameter lines
Message-ID: <498B18A3.6080800@jonkmans.com>

Non-tokenized preprocessor parameter lines

Got me here. Can you elaborate martin?

Thanks!

Matt

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From frank at knobbe.us  Thu Feb  5 13:51:33 2009
From: frank at knobbe.us (Frank Knobbe)
Date: Thu, 05 Feb 2009 12:51:33 -0600
Subject: [Discussion] Submitted Ideas
In-Reply-To: <498A8646.4020902@inliniac.net>
References: <4989ED76.7080703@jonkmans.com>
	<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>
	<498A8646.4020902@inliniac.net>
Message-ID: <1233859893.7327.4.camel@server1>

On Thu, 2009-02-05 at 07:25 +0100, Victor Julien wrote:
> Same here, for all of them though, except the alert message substitution
> and on the fly rule updates...
> 
> It all sounds very interesting... if I only knew what it meant ;-)

alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"GET request
for $VAR1 detected."; content:"GET "; depth:4;
assignvar:offset_4,depth_100,until_space,to_VAR1;)

Request "GET /sumthin" results in alert message: "GET request
for /sumthin detected."

Or something like that... :)

Cheers,
Frank



From lists at inliniac.net  Thu Feb  5 13:59:51 2009
From: lists at inliniac.net (Victor Julien)
Date: Thu, 05 Feb 2009 19:59:51 +0100
Subject: [Discussion] Submitted Ideas
In-Reply-To: <1233859893.7327.4.camel@server1>
References: <4989ED76.7080703@jonkmans.com>	
	<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>	
	<498A8646.4020902@inliniac.net> <1233859893.7327.4.camel@server1>
Message-ID: <498B3727.90600@inliniac.net>

Frank Knobbe wrote:
> On Thu, 2009-02-05 at 07:25 +0100, Victor Julien wrote:
>> Same here, for all of them though, except the alert message substitution
>> and on the fly rule updates...
>>
>> It all sounds very interesting... if I only knew what it meant ;-)
> 
> alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"GET request
> for $VAR1 detected."; content:"GET "; depth:4;
> assignvar:offset_4,depth_100,until_space,to_VAR1;)
> 
> Request "GET /sumthin" results in alert message: "GET request
> for /sumthin detected."
> 
> Or something like that... :)

I got that part, I meant to say I didn't get the points on the list
except this one and the rules reload one :)

In my prototype code I can already capture vars using pcre substring
capturing, although adding an additional way to do this outside of pcre
could be interesting too:

alert tcp any any -> any $HTTP_PORTS (msg:"HTTP GET URI cap";
flow:to_server; content:"GET "; depth:4; pcre:"/^GET
(?P<pkt_http_uri>.*) HTTP\/\d\.\d\r\n/G"; noalert; sid:1;)

This captures the http uri into the var "pkt_http_uri", which is stored
in a packet context. Similarly, "flow_http_uri" stores it in a flow, so
other packets in the same flow can access it. I'm thinking about adding
the same for 'global', 'host', 'stream' and maybe more...

Wrt the alerting, I like this idea, it's pretty simple to implement too.

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From jonkman at jonkmans.com  Thu Feb  5 14:58:38 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Thu, 05 Feb 2009 14:58:38 -0500
Subject: [Discussion] Submitted Ideas
In-Reply-To: <498B3727.90600@inliniac.net>
References: <4989ED76.7080703@jonkmans.com>		<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>		<498A8646.4020902@inliniac.net>
	<1233859893.7327.4.camel@server1> <498B3727.90600@inliniac.net>
Message-ID: <498B44EE.2000001@jonkmans.com>

Victor Julien wrote:
> 
> In my prototype code I can already capture vars using pcre substring
> capturing, although adding an additional way to do this outside of pcre
> could be interesting too:
> 
> alert tcp any any -> any $HTTP_PORTS (msg:"HTTP GET URI cap";
> flow:to_server; content:"GET "; depth:4; pcre:"/^GET
> (?P<pkt_http_uri>.*) HTTP\/\d\.\d\r\n/G"; noalert; sid:1;)
> 
> This captures the http uri into the var "pkt_http_uri", which is stored
> in a packet context. Similarly, "flow_http_uri" stores it in a flow, so
> other packets in the same flow can access it. I'm thinking about adding
> the same for 'global', 'host', 'stream' and maybe more...

With global how do we keep streams from walking over eachother?

Maybe make arrays that the check could be against every element of the
array and if a match then there'd eb the reference to the stream it was
from and further checks could be done for other vars in just that stream...

Matt



-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From martin.fong at sri.com  Thu Feb  5 21:00:53 2009
From: martin.fong at sri.com (Martin Fong)
Date: Thu, 05 Feb 2009 18:00:53 -0800
Subject: [Discussion] Cooperative event loops (e.g.,
 libevent) to support asynch I/O
In-Reply-To: <498B1819.9010304@jonkmans.com>
References: <498B1819.9010304@jonkmans.com>
Message-ID: <498B99D5.4090009@sri.com>

Matt Jonkman wrote:

> Cooperative event loops (e.g., libevent) to support asynch I/O
> 
> Do you mean something like parallel processing of different tools on the
> same stream or event? For instance one stream with http would go through
> regular matching but also be copied out to a http interpreter to pull
> environment variables and the like onto a separate thread/processor?

I'm actually thinking about unix select () loops.  Because the current
implementation simply invokes pcap_loop (), there's no way for a
module to independently perform I/O or respond to a select ()-based
timer.  There have been cases when I've needed to write data to a
socket or read from a pipe, but my preprocessor only got the
opportunity when it "processed" a packet (-- and clearly I don't want
the I/O to block).  (And, yes, I know about the danger of having a
module consume too many cycles doing I/O or computing, but there's
nothing to prevents such abuse now -- and cooperative event processing
_does_ work.)

      As far as your suggestion, that too sounds interesting...!

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090205/16a7d29a/smime.bin

From martin.fong at sri.com  Thu Feb  5 21:11:06 2009
From: martin.fong at sri.com (Martin Fong)
Date: Thu, 05 Feb 2009 18:11:06 -0800
Subject: [Discussion] Non-combinatoric IP/port lists
In-Reply-To: <498B17BD.1080508@jonkmans.com>
References: <498B17BD.1080508@jonkmans.com>
Message-ID: <498B9C3A.2010703@sri.com>

Matt Jonkman wrote:

> Martin, can you elaborate on this one? Not sure what you're getting at.
> 
> Non-combinatoric IP/port lists

Currently, we have blacklist-based rules that look like

     alert tcp [$HOME_NET,!$DNS_SERVERS,!$SMTP_SERVERS] [!$HTTP_PORTS,25] -> [<long IP list>] ...

but clearly the IP/port pairing is combinatoric.  The problem is that
the current rule syntax cannot succinctly express more precise sets of
IP/port bindings without increasing the number of (implicitly
duplicated) rules.  Alternatively I'd like to define some named
IP/port set, and then reference it.  E.g.,

     alert tcp $MY_IP_PORT_BINDING -> [<long IP list>] ...

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090205/6bb59c8f/smime-0001.bin

From martin.fong at sri.com  Thu Feb  5 21:18:46 2009
From: martin.fong at sri.com (Martin Fong)
Date: Thu, 05 Feb 2009 18:18:46 -0800
Subject: [Discussion] Content-based alert message substitution
In-Reply-To: <498B3727.90600@inliniac.net>
References: <4989ED76.7080703@jonkmans.com>		<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>		<498A8646.4020902@inliniac.net>
	<1233859893.7327.4.camel@server1> <498B3727.90600@inliniac.net>
Message-ID: <498B9E06.8000203@sri.com>

Victor Julien wrote:

> Frank Knobbe wrote:
>>
>> alert tcp $EXTERNAL_NET any -> $HOME_NET $HTTP_PORTS (msg:"GET request
>> for $VAR1 detected."; content:"GET "; depth:4;
>> assignvar:offset_4,depth_100,until_space,to_VAR1;)
>>
>> Request "GET /sumthin" results in alert message: "GET request
>> for /sumthin detected."
>>
>> Or something like that... :)
> 
> I got that part, I meant to say I didn't get the points on the list
> except this one and the rules reload one :)
> 
> In my prototype code I can already capture vars using pcre substring
> capturing, although adding an additional way to do this outside of pcre
> could be interesting too:
> 
> alert tcp any any -> any $HTTP_PORTS (msg:"HTTP GET URI cap";
> flow:to_server; content:"GET "; depth:4; pcre:"/^GET
> (?P<pkt_http_uri>.*) HTTP\/\d\.\d\r\n/G"; noalert; sid:1;)
> 
> This captures the http uri into the var "pkt_http_uri", which is stored
> in a packet context. Similarly, "flow_http_uri" stores it in a flow, so
> other packets in the same flow can access it. I'm thinking about adding
> the same for 'global', 'host', 'stream' and maybe more...
> 
> Wrt the alerting, I like this idea, it's pretty simple to implement too.

I've implemented a prototype as a patch and have included some notes
in this e-mail.  The major problem is _where_ to stow derived data.

      Cheers!

      ...Martin

------------------------------------------------------------------------

msg: (parser.c:ParseMessage (char *))

     msg:"*"[, FieldSpecifier]*

FieldSpecifier ::= FieldFormatter '.' FieldName [':' FormatSpecifier]


decode.h:Packet
     EtherHdr eh.ether_{dst,src,type}
     WifiHdr  wifih.{addr[1-4],frame_control,duration_id,seq_control}
     EtherARP ah.ARPHdr ea_hdr.ar_op

     IPHdr    {iph,orig_iph}.ip_{verhl,tos,len,id,off,ttl,proto,csum,src,dst}

     TCPHdr   {tcph,orgi_tcph}.th_{[sd]port,seq,ack,offx2,flags,win,sum,urp}
     u_int8_t *tcp_options_data

     UDPHdr   {udph,orgi_udph}.th_{[sd]port}

     u_int16_t actual_ip_len


preprocessor bhsd_inbound_msg:						\
     "E1[bh] Detected %s %s scan by %s %s of %s IPs: %s",		\
     bhsd.in.ipsweep,							\
     bhsd.in.port_focus,							\
     bhsd.in.scanner_addr,						\
     bhsd.in.SMOI,							\
     bhsd.in.num_IPs							\
     bhsd.in.addresses:ports

preprocessor bhsd_outbound_msg:						\
     "E%s[bh] Detected %s %s port scanning of %s IPs (%s /24s) %s: %s",	\
     bhsd.out.category_type,						\
     bhsd.out.ipsweep,							\
     bhsd.out.port_focus,						\
     bhsd.out.num_IPs,							\
     bhsd.out.num_class_C,						\
     bhsd.out.SMOI,							\
     bhsd.out.ports:counts



alert tcp $EXTERNAL_NET $SHELLCODE_PORTS -> $HOME_NET [135:139,445]      (msg:"My test rule with %s/%s: %s, %s (%s), %s", test.this.label, test.skip.other, 42, Packet.EtherHdr.ether_type:name, Packet.EtherHdr.ether_type, Packet.EtherHdr.ether_src; 
flow:established;  content:"|3131313131313131313131313131313131313131313131|";  classtype:attempted-admin; reference:url,www.eeye.com/html/research/advisories/AD20040501.html; reference:url,www.upenn.edu/computing/virus/04/w32.sasser.worm.html; 
sid:292000032; rev:99; )


11/14-11:29:45.167337  [**] [1:292000032:99] My test rule with this.label/other: 42, IP protocol (0x0800), 00:21:1C:EE:14:00 [**] [Classification: Attempted Administrator Privilege Gain] [Priority: 1] {TCP} 95.28.87.240:21460 -> 192.168.132.9:445
11/14-11:29:47.355986  [**] [1:292000032:99] My test rule with this.label/other: 42, IP protocol (0x0800), 00:21:1C:EE:14:00 [**] [Classification: Attempted Administrator Privilege Gain] [Priority: 1] {TCP} 95.28.87.240:21478 -> 192.168.135.80:445

------------------------------------------------------------------------

MessageFormat								\
     match:"(^My test.+\\)).*";						\
     replace:"\\1, MAC: %s", Packet.EtherHdr.ether_src

MessageFormat								\
     match:"(^E1\\[bh\\].+ IPs):.*";					\
     replace:"\\1: ANONYMIZED"

MessageFormat								\
     match:"^E[58].+";							\
     replace:"\\&, MAC: %s", Packet.EtherHdr.ether_src
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090205/b18adbaa/smime.bin

From martin.fong at sri.com  Thu Feb  5 21:22:34 2009
From: martin.fong at sri.com (Martin Fong)
Date: Thu, 05 Feb 2009 18:22:34 -0800
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <498B186A.7000005@jonkmans.com>
References: <498B186A.7000005@jonkmans.com>
Message-ID: <498B9EEA.8060502@sri.com>

Matt Jonkman wrote:

> Configuration file conditional preprocessor
> 
> Do you mean a preprocessor that'd sanity check or modify the configuration?

The latter: I'd like to have C-preprocessor-like functionality, with
the equivalent of #ifdef, #if defined (), #ifndef, #else, #endif,
etc., statements; and, through -D command-line equivalent symbol
definition, conditional preprocessing of the configuration files.

       Cheers!

       ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090205/498811db/smime.bin

From jonkman at jonkmans.com  Fri Feb  6 10:30:46 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Fri, 06 Feb 2009 10:30:46 -0500
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <498B9EEA.8060502@sri.com>
References: <498B186A.7000005@jonkmans.com> <498B9EEA.8060502@sri.com>
Message-ID: <498C57A6.2060301@jonkmans.com>

So we're only loading certain modules for detection if they are
specifically called for? I.e. don't load the pcre module if there are no
rules asking for pcre?

Matt

Martin Fong wrote:
> Matt Jonkman wrote:
> 
>> Configuration file conditional preprocessor
>>
>> Do you mean a preprocessor that'd sanity check or modify the
>> configuration?
> 
> The latter: I'd like to have C-preprocessor-like functionality, with
> the equivalent of #ifdef, #if defined (), #ifndef, #else, #endif,
> etc., statements; and, through -D command-line equivalent symbol
> definition, conditional preprocessing of the configuration files.
> 
>       Cheers!
> 
>       ...Martin
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From martin.fong at sri.com  Fri Feb  6 13:35:31 2009
From: martin.fong at sri.com (Martin Fong)
Date: Fri, 06 Feb 2009 10:35:31 -0800
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <498C57A6.2060301@jonkmans.com>
References: <498B186A.7000005@jonkmans.com> <498B9EEA.8060502@sri.com>
	<498C57A6.2060301@jonkmans.com>
Message-ID: <498C82F3.4090102@sri.com>

Matt,

> So we're only loading certain modules for detection if they are
> specifically called for? I.e. don't load the pcre module if there are no
> rules asking for pcre?

One specific use case is having two different set of preprocessor
parameters depending on whether the sensor is in front of or behind
a firewall -- this would eliminate the need for building two different,
but mostly identical, configuration files.

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090206/8a59dd00/smime.bin

From martin.fong at sri.com  Fri Feb  6 16:25:11 2009
From: martin.fong at sri.com (Martin Fong)
Date: Fri, 06 Feb 2009 13:25:11 -0800
Subject: [Discussion] Non-tokenized preprocessor parameter lines
In-Reply-To: <498B18A3.6080800@jonkmans.com>
References: <498B18A3.6080800@jonkmans.com>
Message-ID: <498CAAB7.4040206@sri.com>

Matt Jonkman wrote:

> Non-tokenized preprocessor parameter lines

Let me rephrase this into what I'd like (versus definition by
negation): It would be great if processor arguments could (optionally)
_include_ newlines to permit line-oriented parameter definition.  For
example, this would allow

     allow newlines

     preprocessor myPreprocessor:			\
	threshold = 1.0		# a description		\
	max_count = 10		# another description

     disallow newlines

where "[dis]allow newlines" would dictate the parameter token scanner
behavior.

      As a side issue, I'd also like more functionality in the mSplit
() replacement.  Specifically, it would be nice if it accepted 0
(zero) for max_strs and then dynamically allocate the requisite
members, particularly when the input is user-specified and thus
causing the maximum to be relatively unpredictable (e.g., IP
blacklists).

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090206/7a6c7c64/smime.bin

From martin.fong at sri.com  Fri Feb  6 18:07:35 2009
From: martin.fong at sri.com (Martin Fong)
Date: Fri, 06 Feb 2009 15:07:35 -0800
Subject: [Discussion] Submitted Ideas
In-Reply-To: <498B44EE.2000001@jonkmans.com>
References: <4989ED76.7080703@jonkmans.com>		<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>		<498A8646.4020902@inliniac.net>	<1233859893.7327.4.camel@server1>
	<498B3727.90600@inliniac.net> <498B44EE.2000001@jonkmans.com>
Message-ID: <498CC2B7.2000401@sri.com>

Matt Jonkman wrote:

> With global how do we keep streams from walking over each other?
> 
> Maybe make arrays that the check could be against every element of the
> array and if a match then there'd eb the reference to the stream it was
> from and further checks could be done for other vars in just that stream...

Actually, this was the thought behind the "Variable Blackboard" item.
Basically, I think that general problem is the storage of derived or
extracted data -- and this is complicated by namespace and scoping
issues.  For example, can Julien's storage of pcre extracted data into
a packet's context be extended for multiple processors with differing
(opaque) data structures (-- in my field formatter implementation, I
extended the OptTreeNode data structure)?  Where would we store global
versus stream-specific data?

      Could we define a principled and unified approach that would
reduce the complexity and diversity of accessor/setter methods and
storage locations?

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090206/4ab1405a/smime-0001.bin

From lists at inliniac.net  Tue Feb 10 03:44:28 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 09:44:28 +0100
Subject: [Discussion] Submitted Ideas
In-Reply-To: <498B44EE.2000001@jonkmans.com>
References: <4989ED76.7080703@jonkmans.com>		<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>		<498A8646.4020902@inliniac.net>
	<1233859893.7327.4.camel@server1> <498B3727.90600@inliniac.net>
	<498B44EE.2000001@jonkmans.com>
Message-ID: <49913E6C.3070008@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Matt Jonkman wrote:
> Victor Julien wrote:
>> In my prototype code I can already capture vars using pcre substring
>> capturing, although adding an additional way to do this outside of pcre
>> could be interesting too:
>>
>> alert tcp any any -> any $HTTP_PORTS (msg:"HTTP GET URI cap";
>> flow:to_server; content:"GET "; depth:4; pcre:"/^GET
>> (?P<pkt_http_uri>.*) HTTP\/\d\.\d\r\n/G"; noalert; sid:1;)
>>
>> This captures the http uri into the var "pkt_http_uri", which is stored
>> in a packet context. Similarly, "flow_http_uri" stores it in a flow, so
>> other packets in the same flow can access it. I'm thinking about adding
>> the same for 'global', 'host', 'stream' and maybe more...
> 
> With global how do we keep streams from walking over eachother?

You wouldn't. A per flow/stream var would be used then...

A global var would for example just contain the last username that
logged in. Maybe globals aren't very useful for capturing, but more for
stuff like counters, etc.

> Maybe make arrays that the check could be against every element of the
> array and if a match then there'd eb the reference to the stream it was
> from and further checks could be done for other vars in just that stream...

I'm not sure I'm following you here...

Cheers,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmRPmwACgkQiSMBBAuniMfxqACeKe60DF7MSZNBxMiUu83GztL0
6GIAnApP9+GMJR7kC3pH6FkC5HAaxA6F
=qvdf
-----END PGP SIGNATURE-----

From lists at inliniac.net  Tue Feb 10 03:49:48 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 09:49:48 +0100
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <498C82F3.4090102@sri.com>
References: <498B186A.7000005@jonkmans.com>
	<498B9EEA.8060502@sri.com>	<498C57A6.2060301@jonkmans.com>
	<498C82F3.4090102@sri.com>
Message-ID: <49913FAC.9010507@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Fong wrote:
> Matt,
> 
>> So we're only loading certain modules for detection if they are
>> specifically called for? I.e. don't load the pcre module if there are no
>> rules asking for pcre?
> 
> One specific use case is having two different set of preprocessor
> parameters depending on whether the sensor is in front of or behind
> a firewall -- this would eliminate the need for building two different,
> but mostly identical, configuration files.

I understand your goal and I like it. However one of our goals is to
make the configuration & tuning less complex. Adding this type of
complexity could conflict with that goal. On the other hand just having
to configure one config that could be deployed everywhere in your
organization may make it simpler again... thoughts?

Cheers,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmRP6wACgkQiSMBBAuniMdmsQCbBRAuRmJXP++QVye6fBmjHDDY
9UEAn2axA1U+Cuv56931Fi/AUAq4YQIB
=Nbj1
-----END PGP SIGNATURE-----

From lists at inliniac.net  Tue Feb 10 04:12:18 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 10:12:18 +0100
Subject: [Discussion] Submitted Ideas
In-Reply-To: <498CC2B7.2000401@sri.com>
References: <4989ED76.7080703@jonkmans.com>		<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>		<498A8646.4020902@inliniac.net>	<1233859893.7327.4.camel@server1>
	<498B3727.90600@inliniac.net> <498B44EE.2000001@jonkmans.com>
	<498CC2B7.2000401@sri.com>
Message-ID: <499144F2.2090707@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Fong wrote:
> Matt Jonkman wrote:
> 
>> With global how do we keep streams from walking over each other?
>>
>> Maybe make arrays that the check could be against every element of the
>> array and if a match then there'd eb the reference to the stream it was
>> from and further checks could be done for other vars in just that
>> stream...
> 
> Actually, this was the thought behind the "Variable Blackboard" item.
> Basically, I think that general problem is the storage of derived or
> extracted data -- and this is complicated by namespace and scoping
> issues.  For example, can Julien's storage of pcre extracted data into
> a packet's context be extended for multiple processors with differing
> (opaque) data structures (-- in my field formatter implementation, I
> extended the OptTreeNode data structure)?  Where would we store global
> versus stream-specific data?
> 
>      Could we define a principled and unified approach that would
> reduce the complexity and diversity of accessor/setter methods and
> storage locations?

My idea currently is to have variables that can be set, modified and
read from the rule language. The variables should exist in the following
contexts: pkt, host, flow, global.

The variables should be able to have various types: integer, string,
binary (like flowbits in Snort).

The pkt vars are stored in the packet and discarded once the packet is
gone. The host vars are stored in a host structure that is contained in
a host table. It's available as long as the host is. Same for the flow
in the flowtable. The global vars are just in a global table.

With the integers it should able to do some basic calculations, at least
stuff like less than, bigger than etc. Increment/decrement etc. For the
strings I'm thinking about exact matches, starts with, ends with, etc.
Most of this is inspired by ModSecurity that has a number of these
features and that turns out to be very useful.

The way to set/get/modify from rules is an interesting question. I think
we have a separate discussion on rule syntax going on, maybe it should
be a part of that...

Cheers,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmRRPIACgkQiSMBBAuniMcY7ACeNTMGuP01mzm8wmTIfy0xYBvw
bbsAniutjQj7TfjuIqqqDG/zA91gP3te
=9pj8
-----END PGP SIGNATURE-----

From lists at inliniac.net  Tue Feb 10 04:15:21 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 10:15:21 +0100
Subject: [Discussion] Content-based alert message substitution
In-Reply-To: <498B9E06.8000203@sri.com>
References: <4989ED76.7080703@jonkmans.com>		<df9074860902041319l7a35f523i7cde0e84a8aeb98a@mail.gmail.com>		<498A8646.4020902@inliniac.net>	<1233859893.7327.4.camel@server1>
	<498B3727.90600@inliniac.net> <498B9E06.8000203@sri.com>
Message-ID: <499145A9.1020603@inliniac.net>

Martin Fong wrote:
>> Wrt the alerting, I like this idea, it's pretty simple to implement too.
> 
> I've implemented a prototype as a patch and have included some notes
> in this e-mail.  The major problem is _where_ to stow derived data.

In my view the pkt/host/flow vars would be a perfect fit here. The pkt
vars only for vars in the matching packet. But for example a flow var
for getting a username that was captured earlier in the flow...

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From lists at inliniac.net  Tue Feb 10 04:21:09 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 10:21:09 +0100
Subject: [Discussion] Non-tokenized preprocessor parameter lines
In-Reply-To: <498CAAB7.4040206@sri.com>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>
Message-ID: <49914705.5000806@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Fong wrote:
> Matt Jonkman wrote:
> 
>> Non-tokenized preprocessor parameter lines
> 
> Let me rephrase this into what I'd like (versus definition by
> negation): It would be great if processor arguments could (optionally)
> _include_ newlines to permit line-oriented parameter definition.  For
> example, this would allow
> 
>     allow newlines
> 
>     preprocessor myPreprocessor:            \
>     threshold = 1.0        # a description        \
>     max_count = 10        # another description
> 
>     disallow newlines
> 
> where "[dis]allow newlines" would dictate the parameter token scanner
> behavior.
> 
>      As a side issue, I'd also like more functionality in the mSplit
> () replacement.  Specifically, it would be nice if it accepted 0
> (zero) for max_strs and then dynamically allocate the requisite
> members, particularly when the input is user-specified and thus
> causing the maximum to be relatively unpredictable (e.g., IP
> blacklists).

I think we need to work out a rules syntax and configuration scheme
first. I'm not convinced we should have a snort compatible configuration
scheme... I haven't thought of alternatives though.

Regards,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmRRwUACgkQiSMBBAuniMf8JQCbB4nQqxo1UNohtl0+wcAffMDq
VwYAn3yqd2+eKreUVVcmo2+RccVeF4ZR
=Rb/C
-----END PGP SIGNATURE-----

From lists at inliniac.net  Tue Feb 10 04:32:09 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 10:32:09 +0100
Subject: [Discussion] Cooperative event loops (e.g.,
 libevent) to support asynch I/O
In-Reply-To: <498B99D5.4090009@sri.com>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>
Message-ID: <49914999.2000408@inliniac.net>

Martin Fong wrote:
> Matt Jonkman wrote:
> 
>> Cooperative event loops (e.g., libevent) to support asynch I/O
>>
>> Do you mean something like parallel processing of different tools on the
>> same stream or event? For instance one stream with http would go through
>> regular matching but also be copied out to a http interpreter to pull
>> environment variables and the like onto a separate thread/processor?
> 
> I'm actually thinking about unix select () loops.  Because the current
> implementation simply invokes pcap_loop (), there's no way for a
> module to independently perform I/O or respond to a select ()-based
> timer.  There have been cases when I've needed to write data to a
> socket or read from a pipe, but my preprocessor only got the
> opportunity when it "processed" a packet (-- and clearly I don't want
> the I/O to block).  (And, yes, I know about the danger of having a
> module consume too many cycles doing I/O or computing, but there's
> nothing to prevents such abuse now -- and cooperative event processing
> _does_ work.)
> 
>      As far as your suggestion, that too sounds interesting...!

I was thinking about having a way for module to register functions that
will be executed at given times and intervals, probably by having a
separate thread for that. That would introduce a lot of locking issues
of course, so maybe another/additional way is needed.

What about having a module get called at a fixed interval even if there
are no packets? Pcap/ip_queue/nf_queue etc can all work with timeouts or
non-blocking io...

Maybe the module registration functions should be able to determine the
interval, so that if no module requires it the overhead is not imposed,
and if your module does it's set and used.

Interesting...

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From lists at inliniac.net  Tue Feb 10 04:36:56 2009
From: lists at inliniac.net (Victor Julien)
Date: Tue, 10 Feb 2009 10:36:56 +0100
Subject: [Discussion] Non-combinatoric IP/port lists
In-Reply-To: <498B9C3A.2010703@sri.com>
References: <498B17BD.1080508@jonkmans.com> <498B9C3A.2010703@sri.com>
Message-ID: <49914AB8.2030801@inliniac.net>

Martin Fong wrote:
> Matt Jonkman wrote:
> 
>> Martin, can you elaborate on this one? Not sure what you're getting at.
>>
>> Non-combinatoric IP/port lists
> 
> Currently, we have blacklist-based rules that look like
> 
>     alert tcp [$HOME_NET,!$DNS_SERVERS,!$SMTP_SERVERS] [!$HTTP_PORTS,25]
> -> [<long IP list>] ...
> 
> but clearly the IP/port pairing is combinatoric.  The problem is that
> the current rule syntax cannot succinctly express more precise sets of
> IP/port bindings without increasing the number of (implicitly
> duplicated) rules.  

I like this suggestion...

> Alternatively I'd like to define some named
> IP/port set, and then reference it.  E.g.,
> 
>     alert tcp $MY_IP_PORT_BINDING -> [<long IP list>] ...

Interesting too.

Again, something to consider for the configuration & rules syntax...

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From jonkman at jonkmans.com  Tue Feb 10 11:12:27 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Tue, 10 Feb 2009 11:12:27 -0500
Subject: [Discussion] Non-tokenized preprocessor parameter lines
In-Reply-To: <49914705.5000806@inliniac.net>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>
	<49914705.5000806@inliniac.net>
Message-ID: <4991A76B.8030405@jonkmans.com>

I agree, I'm not enamored with the snort-style config. I'd much rather
it be more dynamic, and possibly even real-time adjustable by the engine
to suit it's resources.

Or even better, one that would build a baseline of the box's
capabilities and then config itself to suit. Such as choosing search
methods that fit the ram available, # of threads based on cpu's
available, etc. Take more of this out of black magic guesswork and into
a more scientific method...

Matt

Victor Julien wrote:
> Martin Fong wrote:
>> Matt Jonkman wrote:
> 
>>> Non-tokenized preprocessor parameter lines
>> Let me rephrase this into what I'd like (versus definition by
>> negation): It would be great if processor arguments could (optionally)
>> _include_ newlines to permit line-oriented parameter definition.  For
>> example, this would allow
> 
>>     allow newlines
> 
>>     preprocessor myPreprocessor:            \
>>     threshold = 1.0        # a description        \
>>     max_count = 10        # another description
> 
>>     disallow newlines
> 
>> where "[dis]allow newlines" would dictate the parameter token scanner
>> behavior.
> 
>>      As a side issue, I'd also like more functionality in the mSplit
>> () replacement.  Specifically, it would be nice if it accepted 0
>> (zero) for max_strs and then dynamically allocate the requisite
>> members, particularly when the input is user-specified and thus
>> causing the maximum to be relatively unpredictable (e.g., IP
>> blacklists).
> 
> I think we need to work out a rules syntax and configuration scheme
> first. I'm not convinced we should have a snort compatible configuration
> scheme... I haven't thought of alternatives though.
> 
> Regards,
> Victor
> 

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From mcholste at gmail.com  Tue Feb 10 13:18:02 2009
From: mcholste at gmail.com (Martin Holste)
Date: Tue, 10 Feb 2009 12:18:02 -0600
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <49913FAC.9010507@inliniac.net>
References: <498B186A.7000005@jonkmans.com> <498B9EEA.8060502@sri.com>
	<498C57A6.2060301@jonkmans.com> <498C82F3.4090102@sri.com>
	<49913FAC.9010507@inliniac.net>
Message-ID: <df9074860902101018o158fbf96td55955cbcc0ebd59@mail.gmail.com>

I think that writing and maintaining code capable of interpreting such
dynamic configuration would be cool, but I think that the cost/benefit ratio
isn't there.  True, it would be convenient to have just one config for
everything, but that's only partially true anyway, since each sensor needs
it's own specifics for topology, etc.  That means that you're already doing
per-sensor configuration somewhere along the way, and so all you're really
saving is duplicating config lines.  I just have a few templates lying
around that I use and it seems to work just fine.  It also makes parsing
configs much, much easier when they are declarative and not conditional
(i.e. when you want to create configs via script).

--Martin

On Tue, Feb 10, 2009 at 2:49 AM, Victor Julien <lists at inliniac.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Fong wrote:
> > Matt,
> >
> >> So we're only loading certain modules for detection if they are
> >> specifically called for? I.e. don't load the pcre module if there are no
> >> rules asking for pcre?
> >
> > One specific use case is having two different set of preprocessor
> > parameters depending on whether the sensor is in front of or behind
> > a firewall -- this would eliminate the need for building two different,
> > but mostly identical, configuration files.
>
> I understand your goal and I like it. However one of our goals is to
> make the configuration & tuning less complex. Adding this type of
> complexity could conflict with that goal. On the other hand just having
> to configure one config that could be deployed everywhere in your
> organization may make it simpler again... thoughts?
>
> Cheers,
> Victor
>
> - --
> - ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> - ---------------------------------------------
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkmRP6wACgkQiSMBBAuniMdmsQCbBRAuRmJXP++QVye6fBmjHDDY
> 9UEAn2axA1U+Cuv56931Fi/AUAq4YQIB
> =Nbj1
> -----END PGP SIGNATURE-----
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090210/c1d8a0cb/attachment.html

From mcholste at gmail.com  Tue Feb 10 14:49:36 2009
From: mcholste at gmail.com (Martin Holste)
Date: Tue, 10 Feb 2009 13:49:36 -0600
Subject: [Discussion] Cooperative event loops (e.g.,
	libevent) to support 	asynch I/O
In-Reply-To: <49914999.2000408@inliniac.net>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>
	<49914999.2000408@inliniac.net>
Message-ID: <df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>

What kinds of tasks were you imagining would be executed at intervals?
Garbage collection, statistics?  What did you have in mind?

On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net> wrote:

> Martin Fong wrote:
> > Matt Jonkman wrote:
> >
> >> Cooperative event loops (e.g., libevent) to support asynch I/O
> >>
> >> Do you mean something like parallel processing of different tools on the
> >> same stream or event? For instance one stream with http would go through
> >> regular matching but also be copied out to a http interpreter to pull
> >> environment variables and the like onto a separate thread/processor?
> >
> > I'm actually thinking about unix select () loops.  Because the current
> > implementation simply invokes pcap_loop (), there's no way for a
> > module to independently perform I/O or respond to a select ()-based
> > timer.  There have been cases when I've needed to write data to a
> > socket or read from a pipe, but my preprocessor only got the
> > opportunity when it "processed" a packet (-- and clearly I don't want
> > the I/O to block).  (And, yes, I know about the danger of having a
> > module consume too many cycles doing I/O or computing, but there's
> > nothing to prevents such abuse now -- and cooperative event processing
> > _does_ work.)
> >
> >      As far as your suggestion, that too sounds interesting...!
>
> I was thinking about having a way for module to register functions that
> will be executed at given times and intervals, probably by having a
> separate thread for that. That would introduce a lot of locking issues
> of course, so maybe another/additional way is needed.
>
> What about having a module get called at a fixed interval even if there
> are no packets? Pcap/ip_queue/nf_queue etc can all work with timeouts or
> non-blocking io...
>
> Maybe the module registration functions should be able to determine the
> interval, so that if no module requires it the overhead is not imposed,
> and if your module does it's set and used.
>
> Interesting...
>
> Cheers,
> Victor
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090210/52346cee/attachment.html

From mcholste at gmail.com  Tue Feb 10 14:59:40 2009
From: mcholste at gmail.com (Martin Holste)
Date: Tue, 10 Feb 2009 13:59:40 -0600
Subject: [Discussion] Non-tokenized preprocessor parameter lines
In-Reply-To: <4991A76B.8030405@jonkmans.com>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>
Message-ID: <df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>

I think a fair amount of auto-configuration for the super-techy details
would really help.  To complement that, I'd also really like to see a focus
on performance metrics.  Too often we are in situations where we have to try
to infer something based on rules that _didn't_ fire.  When you're not
confident in a sensor, that's basically impossible.  Some sort of real-time
non-libpcap-based drop statistic or load-shedding would be a huge leap
forward.  For bonus points, a system for providing a 100% objective
performance baseline of a given signature or module would also really help.
I know that each rule performs differently depending on the traffic at hand,
but a metric detailing worst-case/best-case scenario performance would
provide a really nice guideline to aid in making decisions about which rules
should make the cut into the ruleset.  This could be crudely calculated by,
say, the number of PCRE's used, length of content searches, etc.

On Tue, Feb 10, 2009 at 10:12 AM, Matt Jonkman <jonkman at jonkmans.com> wrote:

> I agree, I'm not enamored with the snort-style config. I'd much rather
> it be more dynamic, and possibly even real-time adjustable by the engine
> to suit it's resources.
>
> Or even better, one that would build a baseline of the box's
> capabilities and then config itself to suit. Such as choosing search
> methods that fit the ram available, # of threads based on cpu's
> available, etc. Take more of this out of black magic guesswork and into
> a more scientific method...
>
> Matt
>
> Victor Julien wrote:
> > Martin Fong wrote:
> >> Matt Jonkman wrote:
> >
> >>> Non-tokenized preprocessor parameter lines
> >> Let me rephrase this into what I'd like (versus definition by
> >> negation): It would be great if processor arguments could (optionally)
> >> _include_ newlines to permit line-oriented parameter definition.  For
> >> example, this would allow
> >
> >>     allow newlines
> >
> >>     preprocessor myPreprocessor:            \
> >>     threshold = 1.0        # a description        \
> >>     max_count = 10        # another description
> >
> >>     disallow newlines
> >
> >> where "[dis]allow newlines" would dictate the parameter token scanner
> >> behavior.
> >
> >>      As a side issue, I'd also like more functionality in the mSplit
> >> () replacement.  Specifically, it would be nice if it accepted 0
> >> (zero) for max_strs and then dynamically allocate the requisite
> >> members, particularly when the input is user-specified and thus
> >> causing the maximum to be relatively unpredictable (e.g., IP
> >> blacklists).
> >
> > I think we need to work out a rules syntax and configuration scheme
> > first. I'm not convinced we should have a snort compatible configuration
> > scheme... I haven't thought of alternatives though.
> >
> > Regards,
> > Victor
> >
>
> --
> --------------------------------------------
> Matthew Jonkman
> Emerging Threats
> Phone 765-429-0398
> Fax 312-264-0205
> http://www.emergingthreats.net
> --------------------------------------------
>
> PGP: http://www.jonkmans.com/mattjonkman.asc
>
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090210/723b775f/attachment.html

From jonkman at jonkmans.com  Tue Feb 10 16:05:28 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Tue, 10 Feb 2009 16:05:28 -0500
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <df9074860902101018o158fbf96td55955cbcc0ebd59@mail.gmail.com>
References: <498B186A.7000005@jonkmans.com>
	<498B9EEA.8060502@sri.com>	<498C57A6.2060301@jonkmans.com>
	<498C82F3.4090102@sri.com>	<49913FAC.9010507@inliniac.net>
	<df9074860902101018o158fbf96td55955cbcc0ebd59@mail.gmail.com>
Message-ID: <4991EC18.6080900@jonkmans.com>

Agreed, I don't think one config will serve for an entire net. It still
has to be sensor specific. But hopefully we can make it less complex.

Matt

Martin Holste wrote:
> I think that writing and maintaining code capable of interpreting such
> dynamic configuration would be cool, but I think that the cost/benefit
> ratio isn't there.  True, it would be convenient to have just one config
> for everything, but that's only partially true anyway, since each sensor
> needs it's own specifics for topology, etc.  That means that you're
> already doing per-sensor configuration somewhere along the way, and so
> all you're really saving is duplicating config lines.  I just have a few
> templates lying around that I use and it seems to work just fine.  It
> also makes parsing configs much, much easier when they are declarative
> and not conditional (i.e. when you want to create configs via script).
> 
> --Martin
> 
> On Tue, Feb 10, 2009 at 2:49 AM, Victor Julien <lists at inliniac.net
> <mailto:lists at inliniac.net>> wrote:
> 
> Martin Fong wrote:
>> Matt,
> 
>>> So we're only loading certain modules for detection if they are
>>> specifically called for? I.e. don't load the pcre module if there
> are no
>>> rules asking for pcre?
> 
>> One specific use case is having two different set of preprocessor
>> parameters depending on whether the sensor is in front of or behind
>> a firewall -- this would eliminate the need for building two
> different,
>> but mostly identical, configuration files.
> 
> I understand your goal and I like it. However one of our goals is to
> make the configuration & tuning less complex. Adding this type of
> complexity could conflict with that goal. On the other hand just having
> to configure one config that could be deployed everywhere in your
> organization may make it simpler again... thoughts?
> 
> Cheers,
> Victor
> 
_______________________________________________
Discussion mailing list
Discussion at openinfosecfoundation.org
<mailto:Discussion at openinfosecfoundation.org>
http://lists.openinfosecfoundation.org/mailman/listinfo/discussion

> ------------------------------------------------------------------------

> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From frank at knobbe.us  Tue Feb 10 17:58:11 2009
From: frank at knobbe.us (Frank Knobbe)
Date: Tue, 10 Feb 2009 16:58:11 -0600
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <4991EC18.6080900@jonkmans.com>
References: <498B186A.7000005@jonkmans.com> <498B9EEA.8060502@sri.com>
	<498C57A6.2060301@jonkmans.com> <498C82F3.4090102@sri.com>
	<49913FAC.9010507@inliniac.net>
	<df9074860902101018o158fbf96td55955cbcc0ebd59@mail.gmail.com>
	<4991EC18.6080900@jonkmans.com>
Message-ID: <1234306691.62917.99.camel@server1>

On Tue, 2009-02-10 at 16:05 -0500, Matt Jonkman wrote:
> Agreed, I don't think one config will serve for an entire net. It still
> has to be sensor specific. But hopefully we can make it less complex.

You can ensure that the config uses the "last set option" instead of
throwing an error or using a multiple option. That should allow you to
load a template-based config valid for the whole network (perhaps
http_inspect stuff that's the same across the network), and then loading
sensor specific settings which will override previously configured
template-based defaults so tat you can customize/tweak the settings for
the sensor instance. (We can do that today with variable assignments but
I don't think with other options).

-Frank



From famousjs at gmail.com  Tue Feb 10 20:40:38 2009
From: famousjs at gmail.com (Josh Smith)
Date: Tue, 10 Feb 2009 20:40:38 -0500
Subject: [Discussion] OISF suggestions
Message-ID: <e7646750902101740o79b75068mdcd657951b22e3a8@mail.gmail.com>

I think the administrator should have the ability to sign alerts
created by the OISF engine with PGP.  The administrator could use the
private/public key model so they would be able to tell if the alerts
had been spoofed or altered.

-Josh

From lists at inliniac.net  Wed Feb 11 04:10:57 2009
From: lists at inliniac.net (Victor Julien)
Date: Wed, 11 Feb 2009 10:10:57 +0100
Subject: [Discussion] Cooperative event loops (e.g.,
 libevent) to support 	asynch I/O
In-Reply-To: <df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>	
	<49914999.2000408@inliniac.net>
	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>
Message-ID: <49929621.90109@inliniac.net>

Martin Holste wrote:
> What kinds of tasks were you imagining would be executed at intervals? 
> Garbage collection, statistics?  What did you have in mind?

In Snort_inline I first ran into this when creating the clamav plugin
(not the best of ideas ;-)). I wanted it to check for updated virus defs
on an interval. Like Martin Fong wrote that was done using the packets
timestamp but the code was never activated if there were no packets. Not
a big problem in this case.

Another example is a contract gig I did for a company, where they wanted
a preprocessor to report some stats to a http server on a fixed
interval, no matter if there had been packets or not.

Garbage collection is a good point too. For example cleaning up expired
state... (flow/stream tables come to mind)...

Regards,
Victor

> 
> On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net
> <mailto:lists at inliniac.net>> wrote:
> 
>     Martin Fong wrote:
>     > Matt Jonkman wrote:
>     >
>     >> Cooperative event loops (e.g., libevent) to support asynch I/O
>     >>
>     >> Do you mean something like parallel processing of different tools
>     on the
>     >> same stream or event? For instance one stream with http would go
>     through
>     >> regular matching but also be copied out to a http interpreter to pull
>     >> environment variables and the like onto a separate thread/processor?
>     >
>     > I'm actually thinking about unix select () loops.  Because the current
>     > implementation simply invokes pcap_loop (), there's no way for a
>     > module to independently perform I/O or respond to a select ()-based
>     > timer.  There have been cases when I've needed to write data to a
>     > socket or read from a pipe, but my preprocessor only got the
>     > opportunity when it "processed" a packet (-- and clearly I don't want
>     > the I/O to block).  (And, yes, I know about the danger of having a
>     > module consume too many cycles doing I/O or computing, but there's
>     > nothing to prevents such abuse now -- and cooperative event processing
>     > _does_ work.)
>     >
>     >      As far as your suggestion, that too sounds interesting...!
> 
>     I was thinking about having a way for module to register functions that
>     will be executed at given times and intervals, probably by having a
>     separate thread for that. That would introduce a lot of locking issues
>     of course, so maybe another/additional way is needed.
> 
>     What about having a module get called at a fixed interval even if there
>     are no packets? Pcap/ip_queue/nf_queue etc can all work with timeouts or
>     non-blocking io...
> 
>     Maybe the module registration functions should be able to determine the
>     interval, so that if no module requires it the overhead is not imposed,
>     and if your module does it's set and used.
> 
>     Interesting...
> 
>     Cheers,
>     Victor
> 
>     --
>     ---------------------------------------------
>     Victor Julien
>     http://www.inliniac.net/
>     PGP: http://www.inliniac.net/victorjulien.asc
>     ---------------------------------------------
> 
>     _______________________________________________
>     Discussion mailing list
>     Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>
>     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> 
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From lists at inliniac.net  Wed Feb 11 04:16:14 2009
From: lists at inliniac.net (Victor Julien)
Date: Wed, 11 Feb 2009 10:16:14 +0100
Subject: [Discussion] OISF suggestions
In-Reply-To: <e7646750902101740o79b75068mdcd657951b22e3a8@mail.gmail.com>
References: <e7646750902101740o79b75068mdcd657951b22e3a8@mail.gmail.com>
Message-ID: <4992975E.6060008@inliniac.net>

Josh Smith wrote:
> I think the administrator should have the ability to sign alerts
> created by the OISF engine with PGP.  The administrator could use the
> private/public key model so they would be able to tell if the alerts
> had been spoofed or altered.

I think this is a good suggestion, however I think it should not be part
of the engine itself. I think for alerting we want a setup similar to
Snort's unified->barnyard and I think the pgp stuff can be done in the
barnyard replacement... make sense?

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From lists at inliniac.net  Wed Feb 11 04:27:06 2009
From: lists at inliniac.net (Victor Julien)
Date: Wed, 11 Feb 2009 10:27:06 +0100
Subject: [Discussion] Non-tokenized preprocessor parameter lines
In-Reply-To: <df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>	
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>
	<df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>
Message-ID: <499299EA.1050805@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Holste wrote:
> I think a fair amount of auto-configuration for the super-techy details
> would really help.  To complement that, I'd also really like to see a
> focus on performance metrics.  Too often we are in situations where we
> have to try to infer something based on rules that _didn't_ fire.  When
> you're not confident in a sensor, that's basically impossible.  Some
> sort of real-time non-libpcap-based drop statistic or load-shedding
> would be a huge leap forward.  For bonus points, a system for providing
> a 100% objective performance baseline of a given signature or module
> would also really help.  I know that each rule performs differently
> depending on the traffic at hand, but a metric detailing
> worst-case/best-case scenario performance would provide a really nice
> guideline to aid in making decisions about which rules should make the
> cut into the ruleset.  This could be crudely calculated by, say, the
> number of PCRE's used, length of content searches, etc.

Great suggestion. Matt and I have been talking about doing something
like this for ET sigs for a while already, just never got to actually
building something.

You mentioned that you would like non-libpcap stats. Whats wrong with
them and what is it you want instead?

Regards,
Victor


> On Tue, Feb 10, 2009 at 10:12 AM, Matt Jonkman <jonkman at jonkmans.com
> <mailto:jonkman at jonkmans.com>> wrote:
> 
>     I agree, I'm not enamored with the snort-style config. I'd much rather
>     it be more dynamic, and possibly even real-time adjustable by the engine
>     to suit it's resources.
> 
>     Or even better, one that would build a baseline of the box's
>     capabilities and then config itself to suit. Such as choosing search
>     methods that fit the ram available, # of threads based on cpu's
>     available, etc. Take more of this out of black magic guesswork and into
>     a more scientific method...
> 
>     Matt
> 
>     Victor Julien wrote:
>     > Martin Fong wrote:
>     >> Matt Jonkman wrote:
>     >
>     >>> Non-tokenized preprocessor parameter lines
>     >> Let me rephrase this into what I'd like (versus definition by
>     >> negation): It would be great if processor arguments could
>     (optionally)
>     >> _include_ newlines to permit line-oriented parameter definition.  For
>     >> example, this would allow
>     >
>     >>     allow newlines
>     >
>     >>     preprocessor myPreprocessor:            \
>     >>     threshold = 1.0        # a description        \
>     >>     max_count = 10        # another description
>     >
>     >>     disallow newlines
>     >
>     >> where "[dis]allow newlines" would dictate the parameter token scanner
>     >> behavior.
>     >
>     >>      As a side issue, I'd also like more functionality in the mSplit
>     >> () replacement.  Specifically, it would be nice if it accepted 0
>     >> (zero) for max_strs and then dynamically allocate the requisite
>     >> members, particularly when the input is user-specified and thus
>     >> causing the maximum to be relatively unpredictable (e.g., IP
>     >> blacklists).
>     >
>     > I think we need to work out a rules syntax and configuration scheme
>     > first. I'm not convinced we should have a snort compatible
>     configuration
>     > scheme... I haven't thought of alternatives though.
>     >
>     > Regards,
>     > Victor
>     >
> 
>     --
>     --------------------------------------------
>     Matthew Jonkman
>     Emerging Threats
>     Phone 765-429-0398
>     Fax 312-264-0205
>     http://www.emergingthreats.net
>     --------------------------------------------
> 
>     PGP: http://www.jonkmans.com/mattjonkman.asc
> 
> 
>     _______________________________________________
>     Discussion mailing list
>     Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>
>     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> 
> 


- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmSmeYACgkQiSMBBAuniMeuhwCfdnSPZxC5UG1ITzhhGzfdlhRo
uBEAnRMcybFmg336SyNnQjKm3Ac6EDml
=tl4o
-----END PGP SIGNATURE-----

From mcholste at gmail.com  Wed Feb 11 10:46:38 2009
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 11 Feb 2009 09:46:38 -0600
Subject: [Discussion] Cooperative event loops (e.g.,
	libevent) to support 	asynch I/O
In-Reply-To: <49929621.90109@inliniac.net>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>
	<49914999.2000408@inliniac.net>
	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>
	<49929621.90109@inliniac.net>
Message-ID: <df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>

Ok, well other than the garbage collection, it sounds like all of those are
good candidates for going in the "application layer" detached processes.  As
long as the main detection process allows itself to be reconfigured on the
fly, then there's no reason to bog it down with administrative tasks.  So, I
think the question then becomes one of how do we best allow for a running
configuration to be updated?  Certainly a SIGHUP would be the simplest, but
do we need something less disruptive?

On Wed, Feb 11, 2009 at 3:10 AM, Victor Julien <lists at inliniac.net> wrote:

> Martin Holste wrote:
> > What kinds of tasks were you imagining would be executed at intervals?
> > Garbage collection, statistics?  What did you have in mind?
>
> In Snort_inline I first ran into this when creating the clamav plugin
> (not the best of ideas ;-)). I wanted it to check for updated virus defs
> on an interval. Like Martin Fong wrote that was done using the packets
> timestamp but the code was never activated if there were no packets. Not
> a big problem in this case.
>
> Another example is a contract gig I did for a company, where they wanted
> a preprocessor to report some stats to a http server on a fixed
> interval, no matter if there had been packets or not.
>
> Garbage collection is a good point too. For example cleaning up expired
> state... (flow/stream tables come to mind)...
>
> Regards,
> Victor
>
> >
> > On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net
> > <mailto:lists at inliniac.net>> wrote:
> >
> >     Martin Fong wrote:
> >     > Matt Jonkman wrote:
> >     >
> >     >> Cooperative event loops (e.g., libevent) to support asynch I/O
> >     >>
> >     >> Do you mean something like parallel processing of different tools
> >     on the
> >     >> same stream or event? For instance one stream with http would go
> >     through
> >     >> regular matching but also be copied out to a http interpreter to
> pull
> >     >> environment variables and the like onto a separate
> thread/processor?
> >     >
> >     > I'm actually thinking about unix select () loops.  Because the
> current
> >     > implementation simply invokes pcap_loop (), there's no way for a
> >     > module to independently perform I/O or respond to a select ()-based
> >     > timer.  There have been cases when I've needed to write data to a
> >     > socket or read from a pipe, but my preprocessor only got the
> >     > opportunity when it "processed" a packet (-- and clearly I don't
> want
> >     > the I/O to block).  (And, yes, I know about the danger of having a
> >     > module consume too many cycles doing I/O or computing, but there's
> >     > nothing to prevents such abuse now -- and cooperative event
> processing
> >     > _does_ work.)
> >     >
> >     >      As far as your suggestion, that too sounds interesting...!
> >
> >     I was thinking about having a way for module to register functions
> that
> >     will be executed at given times and intervals, probably by having a
> >     separate thread for that. That would introduce a lot of locking
> issues
> >     of course, so maybe another/additional way is needed.
> >
> >     What about having a module get called at a fixed interval even if
> there
> >     are no packets? Pcap/ip_queue/nf_queue etc can all work with timeouts
> or
> >     non-blocking io...
> >
> >     Maybe the module registration functions should be able to determine
> the
> >     interval, so that if no module requires it the overhead is not
> imposed,
> >     and if your module does it's set and used.
> >
> >     Interesting...
> >
> >     Cheers,
> >     Victor
> >
> >     --
> >     ---------------------------------------------
> >     Victor Julien
> >     http://www.inliniac.net/
> >     PGP: http://www.inliniac.net/victorjulien.asc
> >     ---------------------------------------------
> >
> >     _______________________________________________
> >     Discussion mailing list
> >     Discussion at openinfosecfoundation.org
> >     <mailto:Discussion at openinfosecfoundation.org>
> >     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> >
> >
>
>
> --
> ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> ---------------------------------------------
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090211/d3a830be/attachment-0001.html

From lists at inliniac.net  Wed Feb 11 12:24:24 2009
From: lists at inliniac.net (Victor Julien)
Date: Wed, 11 Feb 2009 18:24:24 +0100
Subject: [Discussion] Cooperative event loops (e.g.,
 libevent) to support 	asynch I/O
In-Reply-To: <df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>	
	<49914999.2000408@inliniac.net>	
	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>	
	<49929621.90109@inliniac.net>
	<df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>
Message-ID: <499309C8.9010605@inliniac.net>

I think the solution of having a time keeping thread that can execute
tasks scheduled by modules would actually lower the overhead. If we
would go for a SIGHUP solution, every part of the system would have to
ask itself "was this for me? Do I have to do something now?" Doing that
in a dedicated very low overhead thread that wakes up only on the times
something needs to be actually done seems more efficient to me.

In my view a HUP would be best for a general "reconfigure yourself", for
example rereading the config and rules files, etc...

Regards,
Victor

Martin Holste wrote:
> Ok, well other than the garbage collection, it sounds like all of those
> are good candidates for going in the "application layer" detached
> processes.  As long as the main detection process allows itself to be
> reconfigured on the fly, then there's no reason to bog it down with
> administrative tasks.  So, I think the question then becomes one of how
> do we best allow for a running configuration to be updated?  Certainly a
> SIGHUP would be the simplest, but do we need something less disruptive?
> 
> On Wed, Feb 11, 2009 at 3:10 AM, Victor Julien <lists at inliniac.net
> <mailto:lists at inliniac.net>> wrote:
> 
>     Martin Holste wrote:
>     > What kinds of tasks were you imagining would be executed at intervals?
>     > Garbage collection, statistics?  What did you have in mind?
> 
>     In Snort_inline I first ran into this when creating the clamav plugin
>     (not the best of ideas ;-)). I wanted it to check for updated virus defs
>     on an interval. Like Martin Fong wrote that was done using the packets
>     timestamp but the code was never activated if there were no packets. Not
>     a big problem in this case.
> 
>     Another example is a contract gig I did for a company, where they wanted
>     a preprocessor to report some stats to a http server on a fixed
>     interval, no matter if there had been packets or not.
> 
>     Garbage collection is a good point too. For example cleaning up expired
>     state... (flow/stream tables come to mind)...
> 
>     Regards,
>     Victor
> 
>     >
>     > On Tue, Feb 10, 2009 at 3:32 AM, Victor Julien <lists at inliniac.net
>     <mailto:lists at inliniac.net>
>     > <mailto:lists at inliniac.net <mailto:lists at inliniac.net>>> wrote:
>     >
>     >     Martin Fong wrote:
>     >     > Matt Jonkman wrote:
>     >     >
>     >     >> Cooperative event loops (e.g., libevent) to support asynch I/O
>     >     >>
>     >     >> Do you mean something like parallel processing of different
>     tools
>     >     on the
>     >     >> same stream or event? For instance one stream with http
>     would go
>     >     through
>     >     >> regular matching but also be copied out to a http
>     interpreter to pull
>     >     >> environment variables and the like onto a separate
>     thread/processor?
>     >     >
>     >     > I'm actually thinking about unix select () loops.  Because
>     the current
>     >     > implementation simply invokes pcap_loop (), there's no way for a
>     >     > module to independently perform I/O or respond to a select
>     ()-based
>     >     > timer.  There have been cases when I've needed to write data
>     to a
>     >     > socket or read from a pipe, but my preprocessor only got the
>     >     > opportunity when it "processed" a packet (-- and clearly I
>     don't want
>     >     > the I/O to block).  (And, yes, I know about the danger of
>     having a
>     >     > module consume too many cycles doing I/O or computing, but
>     there's
>     >     > nothing to prevents such abuse now -- and cooperative event
>     processing
>     >     > _does_ work.)
>     >     >
>     >     >      As far as your suggestion, that too sounds interesting...!
>     >
>     >     I was thinking about having a way for module to register
>     functions that
>     >     will be executed at given times and intervals, probably by
>     having a
>     >     separate thread for that. That would introduce a lot of
>     locking issues
>     >     of course, so maybe another/additional way is needed.
>     >
>     >     What about having a module get called at a fixed interval even
>     if there
>     >     are no packets? Pcap/ip_queue/nf_queue etc can all work with
>     timeouts or
>     >     non-blocking io...
>     >
>     >     Maybe the module registration functions should be able to
>     determine the
>     >     interval, so that if no module requires it the overhead is not
>     imposed,
>     >     and if your module does it's set and used.
>     >
>     >     Interesting...
>     >
>     >     Cheers,
>     >     Victor
>     >
>     >     --
>     >     ---------------------------------------------
>     >     Victor Julien
>     >     http://www.inliniac.net/
>     >     PGP: http://www.inliniac.net/victorjulien.asc
>     >     ---------------------------------------------
>     >
>     >     _______________________________________________
>     >     Discussion mailing list
>     >     Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>
>     >     <mailto:Discussion at openinfosecfoundation.org
>     <mailto:Discussion at openinfosecfoundation.org>>
>     >     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>     >
>     >
> 
> 
>     --
>     ---------------------------------------------
>     Victor Julien
>     http://www.inliniac.net/
>     PGP: http://www.inliniac.net/victorjulien.asc
>     ---------------------------------------------
> 
> 


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From martin.fong at sri.com  Wed Feb 11 13:24:32 2009
From: martin.fong at sri.com (Martin Fong)
Date: Wed, 11 Feb 2009 10:24:32 -0800
Subject: [Discussion]  Plug-Ins
In-Reply-To: <499309C8.9010605@inliniac.net>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>
	<49914999.2000408@inliniac.net>
	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>
	<49929621.90109@inliniac.net>
	<df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>
	<499309C8.9010605@inliniac.net>
Message-ID: <499317E0.8060904@sri.com>

      I'd like to a have a plug-in architecture that simply/only uses
dynamic libraries and configuration files (-- I've _never_ been happy
with the compile-it-into-the-code-base model).

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090211/ddd2194d/smime.bin

From mcholste at gmail.com  Wed Feb 11 15:48:33 2009
From: mcholste at gmail.com (Martin Holste)
Date: Wed, 11 Feb 2009 14:48:33 -0600
Subject: [Discussion] Non-tokenized preprocessor parameter lines
In-Reply-To: <499299EA.1050805@inliniac.net>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>
	<df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>
	<499299EA.1050805@inliniac.net>
Message-ID: <df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>

I've done a little bit of work with rule profiling enough to realize that I
need help from someone who understands Aho-Corasick better so that I can
more accurately figure out what the load would be on the detection engine.
(I posted something to this effect on the Emerging Threats list last week.)
One real performance factor, as far as I can tell, would be to see if the
content matching already appears in another rule.  If that's true, then the
effective load of adding that rule could be negligible, since the detection
engine's effective load doesn't actually increase.  So, a tool which takes
the entire ruleset into account would be very helpful.  I know that
Sourcefire kind of already does this in Snort in a few places, but I think
the number of people who understand the information from the state
transition table reports could be counted on one hand (judging from the lack
of comments on the ET list).  That information needs to be wrapped into a
larger profiler.

Regarding libpcap stats, to put it simply, they lie.  I'm speaking from
Snort experince here, but when I've used router byte counters to audit how
much traffic is going through an interface, then asked Snort how many MB/sec
it processed, the numbers are very, very different until I reduce the load
on the box via subnet-based BPF.  The other problem is that libpcap drop
numbers are completely useless if you're using an Endace DAG card or
(correct me if this is not true) running through iptables.  Undetected drops
have been bad enough in my environment where I've resorted to creating
specific heartbeat signatures and test for the absence of a signature to
detect when a sensor is failing.  That's still far from perfect, though, as
there's plenty of room for drops in the middle.  In any case, that tells a
very different story than asking libpcap how many packets it's dropping.

So here's an idea from left field: what if there were multiple, overlapping
detection engines running which were capable of auditing each other?  It
would be tough to get perfect, but one engine should have about the same TCP
state info as another engine, for instance.  Then a periodic comparison of
those states could shed light on how bad things are.  If the two engines
agree on 99.9 percent of the traffic, it's a safe bet that they are able to
read everything.  Because if they were both overloaded, there's little
chance that they would randomly agree on specific TCP states.  That's just a
brainstorm, and I realize that doubling the load on a box for audit purposes
may be a bit much, but it's that kind of style that I would be looking for,
as opposed to hoping that libpcap is truthful in its reports.

On Wed, Feb 11, 2009 at 3:27 AM, Victor Julien <lists at inliniac.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Holste wrote:
> > I think a fair amount of auto-configuration for the super-techy details
> > would really help.  To complement that, I'd also really like to see a
> > focus on performance metrics.  Too often we are in situations where we
> > have to try to infer something based on rules that _didn't_ fire.  When
> > you're not confident in a sensor, that's basically impossible.  Some
> > sort of real-time non-libpcap-based drop statistic or load-shedding
> > would be a huge leap forward.  For bonus points, a system for providing
> > a 100% objective performance baseline of a given signature or module
> > would also really help.  I know that each rule performs differently
> > depending on the traffic at hand, but a metric detailing
> > worst-case/best-case scenario performance would provide a really nice
> > guideline to aid in making decisions about which rules should make the
> > cut into the ruleset.  This could be crudely calculated by, say, the
> > number of PCRE's used, length of content searches, etc.
>
> Great suggestion. Matt and I have been talking about doing something
> like this for ET sigs for a while already, just never got to actually
> building something.
>
> You mentioned that you would like non-libpcap stats. Whats wrong with
> them and what is it you want instead?
>
> Regards,
> Victor
>
>
> > On Tue, Feb 10, 2009 at 10:12 AM, Matt Jonkman <jonkman at jonkmans.com
> > <mailto:jonkman at jonkmans.com>> wrote:
> >
> >     I agree, I'm not enamored with the snort-style config. I'd much
> rather
> >     it be more dynamic, and possibly even real-time adjustable by the
> engine
> >     to suit it's resources.
> >
> >     Or even better, one that would build a baseline of the box's
> >     capabilities and then config itself to suit. Such as choosing search
> >     methods that fit the ram available, # of threads based on cpu's
> >     available, etc. Take more of this out of black magic guesswork and
> into
> >     a more scientific method...
> >
> >     Matt
> >
> >     Victor Julien wrote:
> >     > Martin Fong wrote:
> >     >> Matt Jonkman wrote:
> >     >
> >     >>> Non-tokenized preprocessor parameter lines
> >     >> Let me rephrase this into what I'd like (versus definition by
> >     >> negation): It would be great if processor arguments could
> >     (optionally)
> >     >> _include_ newlines to permit line-oriented parameter definition.
>  For
> >     >> example, this would allow
> >     >
> >     >>     allow newlines
> >     >
> >     >>     preprocessor myPreprocessor:            \
> >     >>     threshold = 1.0        # a description        \
> >     >>     max_count = 10        # another description
> >     >
> >     >>     disallow newlines
> >     >
> >     >> where "[dis]allow newlines" would dictate the parameter token
> scanner
> >     >> behavior.
> >     >
> >     >>      As a side issue, I'd also like more functionality in the
> mSplit
> >     >> () replacement.  Specifically, it would be nice if it accepted 0
> >     >> (zero) for max_strs and then dynamically allocate the requisite
> >     >> members, particularly when the input is user-specified and thus
> >     >> causing the maximum to be relatively unpredictable (e.g., IP
> >     >> blacklists).
> >     >
> >     > I think we need to work out a rules syntax and configuration scheme
> >     > first. I'm not convinced we should have a snort compatible
> >     configuration
> >     > scheme... I haven't thought of alternatives though.
> >     >
> >     > Regards,
> >     > Victor
> >     >
> >
> >     --
> >     --------------------------------------------
> >     Matthew Jonkman
> >     Emerging Threats
> >     Phone 765-429-0398
> >     Fax 312-264-0205
> >     http://www.emergingthreats.net
> >     --------------------------------------------
> >
> >     PGP: http://www.jonkmans.com/mattjonkman.asc
> >
> >
> >     _______________________________________________
> >     Discussion mailing list
> >     Discussion at openinfosecfoundation.org
> >     <mailto:Discussion at openinfosecfoundation.org>
> >     http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
> >
> >
>
>
> - --
> - ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> - ---------------------------------------------
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkmSmeYACgkQiSMBBAuniMeuhwCfdnSPZxC5UG1ITzhhGzfdlhRo
> uBEAnRMcybFmg336SyNnQjKm3Ac6EDml
> =tl4o
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090211/de45ec59/attachment-0001.html

From martin.fong at sri.com  Wed Feb 11 16:07:37 2009
From: martin.fong at sri.com (Martin Fong)
Date: Wed, 11 Feb 2009 13:07:37 -0800
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <49913FAC.9010507@inliniac.net>
References: <498B186A.7000005@jonkmans.com> <498B9EEA.8060502@sri.com>
	<498C57A6.2060301@jonkmans.com> <498C82F3.4090102@sri.com>
	<49913FAC.9010507@inliniac.net>
Message-ID: <49933E19.5090605@sri.com>

Victor,

 >  I understand your goal and I like it. However one of our goals is
 >  to make the configuration & tuning less complex. Adding this type
 >  of complexity could conflict with that goal. On the other hand just
 >  having to configure one config that could be deployed everywhere in
 >  your organization may make it simpler again... thoughts?

The complexity of a configuration file is subject to its design and
implementation, and I assert that the addition of conditional
processing directives, whose use is optional, neither fosters nor
hinders this complexity.  However, like multiple inheritance in OOPLs,
you'll seldom need it, but when you do, the workarounds are both
inconvenient and ugly.

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090211/9cecc18c/smime.bin

From lists at inliniac.net  Thu Feb 12 03:26:13 2009
From: lists at inliniac.net (Victor Julien)
Date: Thu, 12 Feb 2009 09:26:13 +0100
Subject: [Discussion] rule profiling (was: Re: Non-tokenized preprocessor
 parameter lines)
In-Reply-To: <df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>	
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>	
	<df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>	
	<499299EA.1050805@inliniac.net>
	<df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>
Message-ID: <4993DD25.6050906@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Holste wrote:
> I've done a little bit of work with rule profiling enough to realize
> that I need help from someone who understands Aho-Corasick better so
> that I can more accurately figure out what the load would be on the
> detection engine.  (I posted something to this effect on the Emerging
> Threats list last week.)  One real performance factor, as far as I can
> tell, would be to see if the content matching already appears in another
> rule.  If that's true, then the effective load of adding that rule could
> be negligible, since the detection engine's effective load doesn't
> actually increase.  So, a tool which takes the entire ruleset into
> account would be very helpful.  I know that Sourcefire kind of already
> does this in Snort in a few places, but I think the number of people who
> understand the information from the state transition table reports could
> be counted on one hand (judging from the lack of comments on the ET
> list).  That information needs to be wrapped into a larger profiler.

I'll add in some more variables that matter a lot:

Number of patterns. All pattern matcher algorithms slow down if the
number of patterns increases. This is because the match verification
process gets more expensive as hash tables will fill up (using a 8bit
alphabet in a 2gram algorithm results in a hash of max 65536). Usually
switching from 2gram to 3 (or more) gram versions of algorithms helps,
however at a cost of using more memory & higher computation overhead.
That influences performance quite a bit, probably because of more memory
meaning more CPU cache misses. Usually the average pattern length goes
down too.

The minimum largest pattern size. Most pattern matcher algorithms
perform best with longer patterns since the matcher can step through the
searched data in bigger steps. So if all your sigs have at least a
pattern length of 8 and you add in one of length 3 your performance hit
is going to be bigger than when there are already sigs with smaller
patterns.

Similar rules are grouped together so safe memory on (among others)
pattern matcher contexts. A bad rule will only influence the groups it's
in. But an even worse rule can end up in a lot of contexts. (using the
emerging-all.rules file I can easily have the engine use a few gigabytes
of ram, but using the grouping I have slimmed it down to about 50mb.
Guess which performs better? The smaller one :))

CPU cache size & memory bus speeds seems to make a big difference too.
In my code I've implemented the (simple) BNDM algorithm, both in a
2-gram and 3-gram version. On a Core2duo T5500 (2mb cache) a 2-gram
sBNDM performed best, on Core2duo E6600 (4mb cache) a 3-gram BNDM. On my
gateway box, P3 500mhz 512mb cache, again the 2-gram sBNDM, but with
different hash table sizes and stuff...

One thing that also influences performance is the likelihood of a match
because after the pattern matcher suspects a match, it has to be
verified by the detection engine.  For example I think HTTP keywords,
HTML stuff, SMTP commands, etc, etc, all have a bigger likelihood of
matching and thus are more expensive... maybe a blacklist could help
there. Any pattern on that list would be classified as more expensive...

Regards,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmT3SAACgkQiSMBBAuniMeyRgCfe0nDEaM+qzCpWQ1V2aCfotzp
hOUAn14sw9BI3LZGCRYLKh9dFB6oiubV
=kZSl
-----END PGP SIGNATURE-----

From lists at inliniac.net  Thu Feb 12 03:32:49 2009
From: lists at inliniac.net (Victor Julien)
Date: Thu, 12 Feb 2009 09:32:49 +0100
Subject: [Discussion] stats (was: Re: Non-tokenized preprocessor parameter
 lines)
In-Reply-To: <df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>	
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>	
	<df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>	
	<499299EA.1050805@inliniac.net>
	<df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>
Message-ID: <4993DEB1.4050705@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Holste wrote:
> Regarding libpcap stats, to put it simply, they lie.  I'm speaking from
> Snort experince here, but when I've used router byte counters to audit
> how much traffic is going through an interface, then asked Snort how
> many MB/sec it processed, the numbers are very, very different until I
> reduce the load on the box via subnet-based BPF.  The other problem is
> that libpcap drop numbers are completely useless if you're using an
> Endace DAG card or (correct me if this is not true) running through
> iptables.  Undetected drops have been bad enough in my environment where
> I've resorted to creating specific heartbeat signatures and test for the
> absence of a signature to detect when a sensor is failing.  That's still
> far from perfect, though, as there's plenty of room for drops in the
> middle.  In any case, that tells a very different story than asking
> libpcap how many packets it's dropping.

I wonder how this could be improved. I guess the libpcap stats just
reports drop numbers the kernel gives it. Iptables itself doesn't give
drop numbers I think. Netfilter_queue (the iptables ip_queue
replacement) does, but I don't really know how reliable these numbers are...

Cheers,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmT3rAACgkQiSMBBAuniMcpxwCeIzf+gdQab8AvtYcXYiMVK8XV
VbAAnihSt4wXC/SP0g6FVPfuH+o5hxDF
=pyqE
-----END PGP SIGNATURE-----

From mcholste at gmail.com  Thu Feb 12 19:39:23 2009
From: mcholste at gmail.com (Martin Holste)
Date: Thu, 12 Feb 2009 18:39:23 -0600
Subject: [Discussion] rule profiling (was: Re: Non-tokenized
	preprocessor parameter lines)
In-Reply-To: <4993DD25.6050906@inliniac.net>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>
	<df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>
	<499299EA.1050805@inliniac.net>
	<df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>
	<4993DD25.6050906@inliniac.net>
Message-ID: <df9074860902121639x1ae1ae38n8fdb49c0c2106e87@mail.gmail.com>

Wow, sounds like you're the guy to talk to on this stuff (for the record, I
was counting you as one of the people who understood the state transition
tables).  What do you mean by rule groups?  I'd love to know how you broke
the emerging-all.rules into 50mb.  Running with ac-std takes over 2gb
normally, and I think ac takes well over 4gb, but I've never used it.  Is a
longer match always better?  Is there a threshold at which a pattern (say
100 or so bytes long) is too unwieldy and thus creates a "sweet spot?"

The frequency of matching was kind of what I was getting at the other day on
the ET list regarding the use of the Snort HTTP preproc versus the straight
pattern matcher because I was trying to figure out if the HTTP preproc
(itself having already searched for terms like "POST" and "GET") would allow
us to significantly reduce the load over sigs which use things like
content:"GET"; distance:5;

So, in a brand-new design for a pattern matcher, how can we take advantage
of the fact that we know certain strings will be searched on and hit much
more frequently than others?  Would separating that into a separate thread
provide any advantage?  Perhaps it could become like a mini-barnyard kind of
situation in which it spits out much, much more traffic than alerting but
still only a fraction of the overall throughput. As in, an HTTP preprocessor
that dumps HTTP field streams without doing further app level pattern
matching.  That trims the workload down substantially for another process,
operating barnyard style, to come through and do higher-level matching and
logic.  I think writing to disk barnyard-style would be fairly out of the
question performance-wise, but maybe not writing to a socket where an
entirely different process can read from.  Snort doesn't allow the preproc
to cross CPU's, so the resources are all coming from the same pool.  If your
HTTP preproc had a dedicated CPU, then the cache hits would be extremely
high since it would only search for a few patterns.  If you did it right, I
bet you could get almost ASIC or FPGA-level performance for URI content
searches on a dedicated CPU.

On Thu, Feb 12, 2009 at 2:26 AM, Victor Julien <lists at inliniac.net> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin Holste wrote:
> > I've done a little bit of work with rule profiling enough to realize
> > that I need help from someone who understands Aho-Corasick better so
> > that I can more accurately figure out what the load would be on the
> > detection engine.  (I posted something to this effect on the Emerging
> > Threats list last week.)  One real performance factor, as far as I can
> > tell, would be to see if the content matching already appears in another
> > rule.  If that's true, then the effective load of adding that rule could
> > be negligible, since the detection engine's effective load doesn't
> > actually increase.  So, a tool which takes the entire ruleset into
> > account would be very helpful.  I know that Sourcefire kind of already
> > does this in Snort in a few places, but I think the number of people who
> > understand the information from the state transition table reports could
> > be counted on one hand (judging from the lack of comments on the ET
> > list).  That information needs to be wrapped into a larger profiler.
>
> I'll add in some more variables that matter a lot:
>
> Number of patterns. All pattern matcher algorithms slow down if the
> number of patterns increases. This is because the match verification
> process gets more expensive as hash tables will fill up (using a 8bit
> alphabet in a 2gram algorithm results in a hash of max 65536). Usually
> switching from 2gram to 3 (or more) gram versions of algorithms helps,
> however at a cost of using more memory & higher computation overhead.
> That influences performance quite a bit, probably because of more memory
> meaning more CPU cache misses. Usually the average pattern length goes
> down too.
>
> The minimum largest pattern size. Most pattern matcher algorithms
> perform best with longer patterns since the matcher can step through the
> searched data in bigger steps. So if all your sigs have at least a
> pattern length of 8 and you add in one of length 3 your performance hit
> is going to be bigger than when there are already sigs with smaller
> patterns.
>
> Similar rules are grouped together so safe memory on (among others)
> pattern matcher contexts. A bad rule will only influence the groups it's
> in. But an even worse rule can end up in a lot of contexts. (using the
> emerging-all.rules file I can easily have the engine use a few gigabytes
> of ram, but using the grouping I have slimmed it down to about 50mb.
> Guess which performs better? The smaller one :))
>
> CPU cache size & memory bus speeds seems to make a big difference too.
> In my code I've implemented the (simple) BNDM algorithm, both in a
> 2-gram and 3-gram version. On a Core2duo T5500 (2mb cache) a 2-gram
> sBNDM performed best, on Core2duo E6600 (4mb cache) a 3-gram BNDM. On my
> gateway box, P3 500mhz 512mb cache, again the 2-gram sBNDM, but with
> different hash table sizes and stuff...
>
> One thing that also influences performance is the likelihood of a match
> because after the pattern matcher suspects a match, it has to be
> verified by the detection engine.  For example I think HTTP keywords,
> HTML stuff, SMTP commands, etc, etc, all have a bigger likelihood of
> matching and thus are more expensive... maybe a blacklist could help
> there. Any pattern on that list would be classified as more expensive...
>
> Regards,
> Victor
>
> - --
> - ---------------------------------------------
> Victor Julien
> http://www.inliniac.net/
> PGP: http://www.inliniac.net/victorjulien.asc
> - ---------------------------------------------
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (GNU/Linux)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
>
> iEYEARECAAYFAkmT3SAACgkQiSMBBAuniMeyRgCfe0nDEaM+qzCpWQ1V2aCfotzp
> hOUAn14sw9BI3LZGCRYLKh9dFB6oiubV
> =kZSl
> -----END PGP SIGNATURE-----
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090212/23f7d75f/attachment.html

From lists at inliniac.net  Fri Feb 13 04:36:55 2009
From: lists at inliniac.net (Victor Julien)
Date: Fri, 13 Feb 2009 10:36:55 +0100
Subject: [Discussion] rule profiling
In-Reply-To: <df9074860902121639x1ae1ae38n8fdb49c0c2106e87@mail.gmail.com>
References: <498B18A3.6080800@jonkmans.com> <498CAAB7.4040206@sri.com>	
	<49914705.5000806@inliniac.net> <4991A76B.8030405@jonkmans.com>	
	<df9074860902101159r4ea3e522o2c4dd57944230a04@mail.gmail.com>	
	<499299EA.1050805@inliniac.net>	
	<df9074860902111248q555c4a02t5bb0ac7ff25a8582@mail.gmail.com>	
	<4993DD25.6050906@inliniac.net>
	<df9074860902121639x1ae1ae38n8fdb49c0c2106e87@mail.gmail.com>
Message-ID: <49953F37.8020607@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Holste wrote:
> Wow, sounds like you're the guy to talk to on this stuff (for the
> record, I was counting you as one of the people who understood the state
> transition tables).

I don't consider myself to be an expert, but ever since I'm creating
these things and reading papers about them I'm learning a great deal :)

>  What do you mean by rule groups?  I'd love to know
> how you broke the emerging-all.rules into 50mb.  Running with ac-std
> takes over 2gb normally, and I think ac takes well over 4gb, but I've
> never used it.

In the simplest case we would inspect every packet against every
signature. Thus we would put all patterns in one pattern matcher and
work with that. That doesn't make much sense though, as we know (based
on ports, src and dst settings (think $HOME_NET, etc)) that we're not
interested in many of the sigs. So grouping sigs that are similar makes
sure we inspect less sigs. Example:

alert tcp $HOME_NET any -> any 80 (content:"abc", sid:1)
alert tcp $HOME_NET any -> 1.2.3.4 80 (content:"def", sid:2)
alert tcp $HOME_NET any -> any 80 (content:"ghi", sid:3)

Here we could create 3 sig groups:
tcp, $HOME_NET, any, 0.0.0.0-1.2.3.3:         sids: 1,3
tcp, $HOME_NET, any, 1.2.3.4:                 sids: 1,2,3
tcp, $HOME_NET, any, 1.2.3.5-255.255.255.255: sids: 1,3

When we receive a packet we look up the group that matches. Since we
want to look up only one group, we merge any overlapping group.

In the above exaple sid:2 would only be inspected if the dst addr of a
packet is 1.2.3.4.

This grouping can be taken as far as you want. Protocol, src ip, dst ip,
src port, dst port, dsize, flow status, etc. However, you will easily
end up with many tens of thousands of groups. Each contains a list of
the sigs that will need to be inspected, each has a pattern matcher
context, etc. So that will make memory requirements explode.

While intuitively you'd think it's better to have as many groups as
possible (and thus inspect as little sigs as possible), it doesn't work
that way. I think this is caused by the increased memory usage.

(a lot of memory usage can be reduced by sharing the groups & pattern
matcher contexts. Many groups in practice end up with the exact same
sigs or patterns. Those can share memory. But even then memory usage can
be quite high)

>  Is a longer match always better?  Is there a threshold
> at which a pattern (say 100 or so bytes long) is too unwieldy and thus
> creates a "sweet spot?"

I guess there is a point that longer isn't better, but it's not likely
we'll reach that point in IDS.

> The frequency of matching was kind of what I was getting at the other
> day on the ET list regarding the use of the Snort HTTP preproc versus
> the straight pattern matcher because I was trying to figure out if the
> HTTP preproc (itself having already searched for terms like "POST" and
> "GET") would allow us to significantly reduce the load over sigs which
> use things like content:"GET"; distance:5;

I have no idea about this, I'd be interested in hearing it though.

> So, in a brand-new design for a pattern matcher, how can we take
> advantage of the fact that we know certain strings will be searched on
> and hit much more frequently than others?

One thing I'm thinking about is to group expensive sigs together as much
as possible, so the other groups won't suffer from them...

>  Would separating that into a
> separate thread provide any advantage?

Interesting idea. Maybe we can get the expensive sig groups to be
handled by a different thread than the others. I have to think about this...

>  Perhaps it could become like a
> mini-barnyard kind of situation in which it spits out much, much more
> traffic than alerting but still only a fraction of the overall
> throughput. As in, an HTTP preprocessor that dumps HTTP field streams
> without doing further app level pattern matching.  That trims the
> workload down substantially for another process, operating barnyard
> style, to come through and do higher-level matching and logic.  I think
> writing to disk barnyard-style would be fairly out of the question
> performance-wise, but maybe not writing to a socket where an entirely
> different process can read from.  Snort doesn't allow the preproc to
> cross CPU's, so the resources are all coming from the same pool.  If
> your HTTP preproc had a dedicated CPU, then the cache hits would be
> extremely high since it would only search for a few patterns.  If you
> did it right, I bet you could get almost ASIC or FPGA-level performance
> for URI content searches on a dedicated CPU.

One thing I've been dreaming about is having something like ModSecurity
be able to take data from us, maybe using a socket or pipe. That way it
could be a separate process or maybe even be running on a separate box...

Regards,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmVPzUACgkQiSMBBAuniMctjACfb2fTa0BKWCb3YB6V1y+aEpaG
+uEAnjdLVfC4U1TQjOWe0rVasjA1gTBh
=6Nna
-----END PGP SIGNATURE-----

From lists at inliniac.net  Fri Feb 13 04:39:18 2009
From: lists at inliniac.net (Victor Julien)
Date: Fri, 13 Feb 2009 10:39:18 +0100
Subject: [Discussion] Plug-Ins
In-Reply-To: <499317E0.8060904@sri.com>
References: <498B1819.9010304@jonkmans.com>
	<498B99D5.4090009@sri.com>	<49914999.2000408@inliniac.net>	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>	<49929621.90109@inliniac.net>	<df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>	<499309C8.9010605@inliniac.net>
	<499317E0.8060904@sri.com>
Message-ID: <49953FC6.1020102@inliniac.net>

Martin Fong wrote:
>      I'd like to a have a plug-in architecture that simply/only uses
> dynamic libraries and configuration files (-- I've _never_ been happy
> with the compile-it-into-the-code-base model).

Yeah, this is something I want as well. I wonder how far we should take
this though: detection modules only, or also output, decoding, pattern
matcher, packet sources (libpcap, ip_queue, etc), etc?

Thoughts?

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From lists at inliniac.net  Fri Feb 13 04:44:58 2009
From: lists at inliniac.net (Victor Julien)
Date: Fri, 13 Feb 2009 10:44:58 +0100
Subject: [Discussion] Configuration file conditional preprocessor
In-Reply-To: <49933E19.5090605@sri.com>
References: <498B186A.7000005@jonkmans.com> <498B9EEA.8060502@sri.com>
	<498C57A6.2060301@jonkmans.com> <498C82F3.4090102@sri.com>
	<49913FAC.9010507@inliniac.net> <49933E19.5090605@sri.com>
Message-ID: <4995411A.6080309@inliniac.net>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin Fong wrote:
> Victor,
> 
>>  I understand your goal and I like it. However one of our goals is
>>  to make the configuration & tuning less complex. Adding this type
>>  of complexity could conflict with that goal. On the other hand just
>>  having to configure one config that could be deployed everywhere in
>>  your organization may make it simpler again... thoughts?
> 
> The complexity of a configuration file is subject to its design and
> implementation, and I assert that the addition of conditional
> processing directives, whose use is optional, neither fosters nor
> hinders this complexity.  However, like multiple inheritance in OOPLs,
> you'll seldom need it, but when you do, the workarounds are both
> inconvenient and ugly.

You're right about that Martin, I just fear it will increase
implementation complexity quite a bit for just a very small group of
users. But maybe it's worth it.

Does anyone have ideas about what kind of configuration format to use? I
think it should be:
- - easy to edit by humans
- - easy to parse by programs
- - preferably be (similar to) something ppl already know

I think Snort3 is going to use Lua for example. ModSecurity uses Apache
style configuration.

Regards,
Victor

- --
- ---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
- ---------------------------------------------

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkmVQRoACgkQiSMBBAuniMcddACeOxnS1WmFpO9XWoaua5Yj7FuP
FHQAnAkSrnzTCIYAchePywpd1TE6lcHk
=qOiC
-----END PGP SIGNATURE-----

From martin.fong at sri.com  Fri Feb 13 13:23:41 2009
From: martin.fong at sri.com (Martin Fong)
Date: Fri, 13 Feb 2009 10:23:41 -0800
Subject: [Discussion] Plug-Ins
In-Reply-To: <49953FC6.1020102@inliniac.net>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>
	<49914999.2000408@inliniac.net>
	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>
	<49929621.90109@inliniac.net>
	<df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>
	<499309C8.9010605@inliniac.net> <499317E0.8060904@sri.com>
	<49953FC6.1020102@inliniac.net>
Message-ID: <4995BAAD.4070103@sri.com>

Victor,

 >>  I'd like to a have a plug-in architecture that simply/only uses
 >>  dynamic libraries and configuration files (-- I've _never_ been
 >>  happy with the compile-it-into-the-code-base model).
 >
 >  Yeah, this is something I want as well. I wonder how far we should
 >  take this though: detection modules only, or also output, decoding,
 >  pattern matcher, packet sources (libpcap, ip_queue, etc), etc?

I vote for all of them.  This would eliminate the fragility associated
with patching configure[,.in] and Makefile[.ac,.in] scripts (which are
highly subject to the versions of autoconf and automake), and it's
inexpensive, because address reconciliation is only done once.

      Cheers!

      ...Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5193 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090213/cf5626ad/smime.bin

From jonkman at jonkmans.com  Sat Feb 14 10:39:43 2009
From: jonkman at jonkmans.com (Matt Jonkman)
Date: Sat, 14 Feb 2009 10:39:43 -0500
Subject: [Discussion] Plug-Ins
In-Reply-To: <4995BAAD.4070103@sri.com>
References: <498B1819.9010304@jonkmans.com>
	<498B99D5.4090009@sri.com>	<49914999.2000408@inliniac.net>	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>	<49929621.90109@inliniac.net>	<df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>	<499309C8.9010605@inliniac.net>
	<499317E0.8060904@sri.com>	<49953FC6.1020102@inliniac.net>
	<4995BAAD.4070103@sri.com>
Message-ID: <4996E5BF.2020905@jonkmans.com>

I third that, loadable plugins would definitely make the compile and
customization process MUCH easier than preprocessors and all.

Matt

Martin Fong wrote:
> Victor,
> 
>>>  I'd like to a have a plug-in architecture that simply/only uses
>>>  dynamic libraries and configuration files (-- I've _never_ been
>>>  happy with the compile-it-into-the-code-base model).
>>
>>  Yeah, this is something I want as well. I wonder how far we should
>>  take this though: detection modules only, or also output, decoding,
>>  pattern matcher, packet sources (libpcap, ip_queue, etc), etc?
> 
> I vote for all of them.  This would eliminate the fragility associated
> with patching configure[,.in] and Makefile[.ac,.in] scripts (which are
> highly subject to the versions of autoconf and automake), and it's
> inexpensive, because address reconciliation is only done once.
> 
>      Cheers!
> 
>      ...Martin
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc



From ali.ikinci at contentkeeper.com  Wed Feb 25 09:52:28 2009
From: ali.ikinci at contentkeeper.com (Ali Ikinci)
Date: Wed, 25 Feb 2009 16:52:28 +0200
Subject: [Discussion] The Revealer a proposal for the solution to many of
	our problems
Message-ID: <1235573548.9992.157.camel@enterprise>

Before I present my proposal I want to say that I am a relatively new
security researcher and a software engineer. I am working on my own
client honeypot (http://monkeyspider.sf.net) and am not so well read on
IDS/IPS. I am writing this down in a kind of brainstorming session of
mine over a few days and while reading the list archive and taking
notes, so don't take anything here too serious. It might contain a dozen
of mistakes and might not be well formed.

In this post I do not want to go into too much detail about detection
mechanisms and features of our system as there are already many great
ideas from real pros in this list. There is no doubt that these ideas
should all be cherished in some or the other way. While doing this I
think we have the problem of not seeing the forest for the trees. We
know what we need to do and how we can detect threats and collect IP/DNS
reputation data but not how we can find a home for all this ideas while
keeping it simple to assure a wide use and profit from many sensors
worldwide. That's where this proposal comes in.

I am thinking of a way how we can bundle all ideas, avoid duplicate
work, contribute to the community, build a (the) big IP/DNS reputation
system, invent/start new software projects/frameworks, make use of all
available systems, keep it simple for us and the users and be not
dependent on a vendor or authority..

So here are some trade-offs which I think we have to deal with:

Not 'reinventing the Wheel' vs. Developing something new
There are already many good software packages/frameworks who are very
good at a particular matter and already have a great pool of reputation
data. Why not use all of them (maybe only for one purpose) than we get a
system which is good at all minor aspects and features. We can at
anytime contribute to the projects or start our own sub projects with
features and functionality we want to add.

A quote from John Pritchard's post:
"I'd like to suggest that we take very large deployments into
consideration when designing our solution. The kind of problems you
encounter when managing an infrastructure with a handful or even a
dozen different IDS sensors is very different than trying to
efficiently and consistently manage infrastructures with larger
footprints (e.g. > 100+ sensors)."

Scalability and many features vs. Complexity for granny and her grandson
When we need to deploy a system with so many features and such a
complexity which involves the latest threats and detection techniques
the vast majority of people including many well read sysadmins will have
it hard to implement and configure all features of this system and even
harder to scale this to more than one system. Whereas we need to have as
much users as we can get to have more sensors out there waiting and
delivering reputation data to us and thus to the whole community.

Centralization vs. Distribution
Do we need a central instance to store data? What if that instance gets
bought up by a company or gets otherwise compromised.

A New Signature/Policy format vs. various available signature/policy
formats:
There are already a wide variety of tools and formats. If we use only
one of these tools then we will need to adopt their format and will also
inherit all their shortcomings. Like in this discussion has been about
snort and bro. Whereas if we use multiple programs we need to administer
multiple formats. If we build a new format than we can not profit from
the reputation data which is already there or we have to convert in some
or the other way.

Forking projects vs. Rewriting similar projects vs. Extension of
available projects: 
Forking is bad and only a last solution to an otherwise unresolvable
problem. Free projects normally remain to be free for their whole life
so if we want to have new features we should try and add them to the
existing projects. Only customizations to our system can be written as
sub projects.

Proposal: "The Revealer"

Contrary to a software package/framework as the outcome of this
discussion: A custom full-fledged Ubuntu-Server edition customized to
and configured from the start to be part of a master-slave P2P network
of OISF nodes called 'the Revealer' (or Seeker) (One could take ideas
out of systems like Roo but also IPCop,m0n0wall etc.):

Featuring many available solutions for various detection/categorization
fields bundeled with monitoring and management interfaces.

This system could operate as a Layer2 bridge(or a network tap? or as a
router?).

Next Gen ideas and technologies should be built-in to be long lived,
usable and up-to date like Virtualization, P2P, Web 2.0, Honeypots/nets,
Cloud Computing etc.

We would then add sub projects(for example a new ids system) which might
also support plug-ins. These should be self-contained free-software
projects so that they can be reused in other projects/deployments.

The Revealer distribution should be modular and modules (like
snort,bro,our ids etc.) maybe de/activated during runtime with a click
on the web interface. These modules would come with a good standard
configuration and submission setting.

There would be one central Monitoring and Management web interface which
would be located at the master server. Including centralized patch and
signature management. And many slaves connected and would be
administered from the master.

A new meta policy format which would become a sort of a database of what
is already there for bro and snort and which would also contain its own
information regarding the signatures. For example to have a snort
signature together with the human readable information about who has
written it and how it was detected etc. This new policy format should
also be used as the main policy format for other detection (sub)projects
which we write ourselves. Think of the CVE database.

Finally we would get the best database of threats and contribute heavily
to the security of the net as a whole. Global culprits would be
identified and banned accurately and without a payment/subscription to a
third party!

Further ideas regarding the Revealer:

I.) Redirecting crackers to server honeypots: We could also
automatically redirect attackers trying to crack into remote peers to
our local honeypots or other researchers honeynets and thus collect all
crackers/malware at well observed and researched honeynets. This could
be done using deception techniques to trick the attacker to believe they
are still attacking the same ip so all traffic from the attacker should
be routed over the initially attacked node.

II.) Redirecting malicious traffic to client honeypots: We could
built-in a mechanism to gather all kind of interesting file types out of
a network flow (this has already been suggested but we need client
honeypot detection mechanisms in addition to AV signatures) like
executables, JavaScript, ActiveX, VBS, PDF, SWF, OLE etc. which could be
malicious to the requesting clients and hand them over to various
AV-Solutions,client honeypots (like Caffeine Monkey, phoneyc, honeyc
etc.) and CWSandbox. Some of these could directly run on the Revealer
itself.

III.) A simple user contribution interface at the main web interface of
each master node. This would enable the user to submit various
threats/malicious files/suspicious things etc. So she could use this
simple interface to anonymously submit suspicious files to us (and/or to
CWSandbox, Virustotal) or URL's or comments about suspicious behavior
etc. This is something which might become a powerful feature. Maybe also
a place to submit self written snort/bro signatures and the like. A
widespread and active community means power.

IV.) Built-in support for KVM based Ubuntu JeOS installations for
example as a containment strategy for honeypots etc. 

Some reasons for the design decisions:

A.) Automated deployment, configure once deploy 100x times.
http://www.ubuntu.com/products/whatisubuntu/serveredition/features/autodeploy

B.) Full-fledged OS vs. A software package/framework:
i. The question is not an either/or but an also. We are going to have a
software package/framework with many sub projects etc. and this will be
deployed in a full-fledged OS environment.
ii. Why it is a reasonable overhead to have a full operating system
rather than only a software package/framework? We do not have to take
care about much with the operating system. 

C.) Decentralized Peer-to-Peer network with a two tier structure: 
i. Masters are part of the p2p network slaves are not.
ii. Slaves and Masters communicate over secured RPC calls.
iii. There are two kinds of peers. Nodes and Supernodes. Nodes would
only contribute to and profit from the central IP Data. Supernodes would
also hold and distribute the distributed data(base). The following
diagram gives an idea of the structure:
http://ikinci.info/oisf-proposal.png   

Reasons for the P2P architecture:
1. Distributed database.
2. Nearly instant updates. P2P is the fastest way of distributing data.
3. P2P is the most hack-resistant architecture till date.
4. Independent from any vendor and thus always free signatures

Further advantages of this approach:

a) Scalability: from a home system to hundreds of systems connected to
each other in a master slave configuration. Efficiently and consistently
managing hundreds of distributed sensors worldwide! 

b) Virtualization and Containment Strategy: Are already built-ins from
Ubuntu see JeOS Linux KVMs and App Armor.
http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos
http://www.ubuntu.com/products/whatisubuntu/serveredition/features/security
Our system could be deployed on any PC with VMWare Player for example
where we would only provide a ready to run full configured VMWare Image
with the JeOS so that anyone could use our system with a couple of mouse
clicks to protect only their windows pc connected to the net.

c) Support for many (scripting) languages (c, c++, fortran, perl,
python, ruby, tcl etc.). We don't have to discourage capable developers/
contributors only because we don't support their favorite programming
language. And we can make use of the vast libraries and tools of these.
Take a look at the many small scripts and tools in many different
programming languages inside of BackTrack.

d) Built-in Security and software management with a big pool of actively
maintained software. We don't have to have any software support for
packages which are already part of the big Ubuntu repositories and only
need to have updates and ports for our custom packages or special
versions of our used tools. We therefore can use the development
versions of the package maintainers etc. And we only had to provide an
additional repository server with our packages on it.

e) Easy deployment, straightforward installation and out-of-the-box
standard configuration. Maybe this is not for granny but her grandson
could install this on an old spare pc or on a virtual machine on
granny's laptop. Dual core laptops are already standard. But I think our
system will be of most use for deployments who have at least a handful
of PC's and a system admin.

f) Easy integration and administration of other packages for example
Monitoring software like Nagios etc. 

g) Easy administration due to a standard web interface where features
can be turned on and off and any further configuration would still be
able over a shell and the applications configuration files. We could
also support the installation of other maintenance tools like Webmin
etc. so that the user gets powerful administration options and we don't
have to write too much.

h) Easy additions to the standard distribution as the system is a
standard Ubuntu/Debian installation.

i) Easy integration of honeypot/honeynet technologies: honeypots, client
honeypots etc. for example nepenthes, honeyd etc.

j) Standardized release cycles of Ubuntu and guaranteed maintenance
period.

k) passive OS fingerprinting (p0f and others)

We could start slow and implement features over time. The first version
could be built in a couple of days! We could start with the approach of
reusing software, and writing new software/glue code should only be the
last resort.

Thanks for your attention and your precious time

Ali


From lists at inliniac.net  Sat Feb 28 09:14:20 2009
From: lists at inliniac.net (Victor Julien)
Date: Sat, 28 Feb 2009 15:14:20 +0100
Subject: [Discussion] Plug-Ins
In-Reply-To: <4995BAAD.4070103@sri.com>
References: <498B1819.9010304@jonkmans.com> <498B99D5.4090009@sri.com>
	<49914999.2000408@inliniac.net>
	<df9074860902101149q8477bf5s24f7cc8abfc8b210@mail.gmail.com>
	<49929621.90109@inliniac.net>
	<df9074860902110746y2f306b92kd4dac3d397735722@mail.gmail.com>
	<499309C8.9010605@inliniac.net> <499317E0.8060904@sri.com>
	<49953FC6.1020102@inliniac.net> <4995BAAD.4070103@sri.com>
Message-ID: <49A946BC.5020803@inliniac.net>

Martin Fong wrote:
> Victor,
> 
>>>  I'd like to a have a plug-in architecture that simply/only uses
>>>  dynamic libraries and configuration files (-- I've _never_ been
>>>  happy with the compile-it-into-the-code-base model).
>>
>>  Yeah, this is something I want as well. I wonder how far we should
>>  take this though: detection modules only, or also output, decoding,
>>  pattern matcher, packet sources (libpcap, ip_queue, etc), etc?
> 
> I vote for all of them.  This would eliminate the fragility associated
> with patching configure[,.in] and Makefile[.ac,.in] scripts (which are
> highly subject to the versions of autoconf and automake), and it's
> inexpensive, because address reconciliation is only done once.

I agree, let's do as many as possible.

Btw, about the automake stuff, in another project I'm testing a build
system called 'waf' (http://code.google.com/p/waf/). I replaces
configure, automake, autoconf & make and only depends on python. So far
I'm really liking it. It's cool to have a build system that is actually
understandable. If it works out in my other project, I'm strongly
considering it for OISF as well.

Cheers,
Victor

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From lists at inliniac.net  Sat Feb 28 09:34:41 2009
From: lists at inliniac.net (Victor Julien)
Date: Sat, 28 Feb 2009 15:34:41 +0100
Subject: [Discussion] The Revealer a proposal for the solution to many
 of	our problems
In-Reply-To: <1235573548.9992.157.camel@enterprise>
References: <1235573548.9992.157.camel@enterprise>
Message-ID: <49A94B81.30504@inliniac.net>

Hi Ali,

In general I'd like to say: very interesting ideas, but I think most are
outside of the scope of this project. What we're building here is an
actual detection engine.

Further comments inline:

Ali Ikinci wrote:
<snip>
> I am thinking of a way how we can bundle all ideas, avoid duplicate
> work, contribute to the community, build a (the) big IP/DNS reputation
> system, invent/start new software projects/frameworks, make use of all
> available systems, keep it simple for us and the users and be not
> dependent on a vendor or authority..

The independence is the reason we're organized as a non-profit.

> So here are some trade-offs which I think we have to deal with:
> 
> Not 'reinventing the Wheel' vs. Developing something new
> There are already many good software packages/frameworks who are very
> good at a particular matter and already have a great pool of reputation
> data. Why not use all of them (maybe only for one purpose) than we get a
> system which is good at all minor aspects and features. We can at
> anytime contribute to the projects or start our own sub projects with
> features and functionality we want to add.
> 
> A quote from John Pritchard's post:
> "I'd like to suggest that we take very large deployments into
> consideration when designing our solution. The kind of problems you
> encounter when managing an infrastructure with a handful or even a
> dozen different IDS sensors is very different than trying to
> efficiently and consistently manage infrastructures with larger
> footprints (e.g. > 100+ sensors)."
> 
> Scalability and many features vs. Complexity for granny and her grandson
> When we need to deploy a system with so many features and such a
> complexity which involves the latest threats and detection techniques
> the vast majority of people including many well read sysadmins will have
> it hard to implement and configure all features of this system and even
> harder to scale this to more than one system. Whereas we need to have as
> much users as we can get to have more sensors out there waiting and
> delivering reputation data to us and thus to the whole community.

I think our engine should work as much as possible "out of box".
Additionally I think it should be suitable for being configured by
external programs like frontends etc.

> Centralization vs. Distribution
> Do we need a central instance to store data? What if that instance gets
> bought up by a company or gets otherwise compromised.

I don't think OISF is actually going to maintain the reputation lists,
that is maybe more something for Emerging Threats and other 3rd parties.
For legal reasons too, since we don't want to be able to be sued for
listing someone and risking losing the development funds.

I think our engine should be able to use an existing reputation data
format standard, or develop one.

> A New Signature/Policy format vs. various available signature/policy
> formats:
> There are already a wide variety of tools and formats. If we use only
> one of these tools then we will need to adopt their format and will also
> inherit all their shortcomings. Like in this discussion has been about
> snort and bro. Whereas if we use multiple programs we need to administer
> multiple formats. If we build a new format than we can not profit from
> the reputation data which is already there or we have to convert in some
> or the other way.

My personal preference would be to use the Snort format as many people
out there already know it. We may have to extend it in some ways, making
it non standard.

> Forking projects vs. Rewriting similar projects vs. Extension of
> available projects: 
> Forking is bad and only a last solution to an otherwise unresolvable
> problem. Free projects normally remain to be free for their whole life
> so if we want to have new features we should try and add them to the
> existing projects. Only customizations to our system can be written as
> sub projects.

The engine we're building is going to be a from scratch implementation.

"The Revealer" proposal below I think falls outside of our scope. It
does sound like a great idea for a project though!

Cheers,
Victor

> Proposal: "The Revealer"
> 
> Contrary to a software package/framework as the outcome of this
> discussion: A custom full-fledged Ubuntu-Server edition customized to
> and configured from the start to be part of a master-slave P2P network
> of OISF nodes called 'the Revealer' (or Seeker) (One could take ideas
> out of systems like Roo but also IPCop,m0n0wall etc.):
> 
> Featuring many available solutions for various detection/categorization
> fields bundeled with monitoring and management interfaces.
> 
> This system could operate as a Layer2 bridge(or a network tap? or as a
> router?).
> 
> Next Gen ideas and technologies should be built-in to be long lived,
> usable and up-to date like Virtualization, P2P, Web 2.0, Honeypots/nets,
> Cloud Computing etc.
> 
> We would then add sub projects(for example a new ids system) which might
> also support plug-ins. These should be self-contained free-software
> projects so that they can be reused in other projects/deployments.
> 
> The Revealer distribution should be modular and modules (like
> snort,bro,our ids etc.) maybe de/activated during runtime with a click
> on the web interface. These modules would come with a good standard
> configuration and submission setting.
> 
> There would be one central Monitoring and Management web interface which
> would be located at the master server. Including centralized patch and
> signature management. And many slaves connected and would be
> administered from the master.
> 
> A new meta policy format which would become a sort of a database of what
> is already there for bro and snort and which would also contain its own
> information regarding the signatures. For example to have a snort
> signature together with the human readable information about who has
> written it and how it was detected etc. This new policy format should
> also be used as the main policy format for other detection (sub)projects
> which we write ourselves. Think of the CVE database.
> 
> Finally we would get the best database of threats and contribute heavily
> to the security of the net as a whole. Global culprits would be
> identified and banned accurately and without a payment/subscription to a
> third party!
> 
> Further ideas regarding the Revealer:
> 
> I.) Redirecting crackers to server honeypots: We could also
> automatically redirect attackers trying to crack into remote peers to
> our local honeypots or other researchers honeynets and thus collect all
> crackers/malware at well observed and researched honeynets. This could
> be done using deception techniques to trick the attacker to believe they
> are still attacking the same ip so all traffic from the attacker should
> be routed over the initially attacked node.
> 
> II.) Redirecting malicious traffic to client honeypots: We could
> built-in a mechanism to gather all kind of interesting file types out of
> a network flow (this has already been suggested but we need client
> honeypot detection mechanisms in addition to AV signatures) like
> executables, JavaScript, ActiveX, VBS, PDF, SWF, OLE etc. which could be
> malicious to the requesting clients and hand them over to various
> AV-Solutions,client honeypots (like Caffeine Monkey, phoneyc, honeyc
> etc.) and CWSandbox. Some of these could directly run on the Revealer
> itself.
> 
> III.) A simple user contribution interface at the main web interface of
> each master node. This would enable the user to submit various
> threats/malicious files/suspicious things etc. So she could use this
> simple interface to anonymously submit suspicious files to us (and/or to
> CWSandbox, Virustotal) or URL's or comments about suspicious behavior
> etc. This is something which might become a powerful feature. Maybe also
> a place to submit self written snort/bro signatures and the like. A
> widespread and active community means power.
> 
> IV.) Built-in support for KVM based Ubuntu JeOS installations for
> example as a containment strategy for honeypots etc. 
> 
> Some reasons for the design decisions:
> 
> A.) Automated deployment, configure once deploy 100x times.
> http://www.ubuntu.com/products/whatisubuntu/serveredition/features/autodeploy
> 
> B.) Full-fledged OS vs. A software package/framework:
> i. The question is not an either/or but an also. We are going to have a
> software package/framework with many sub projects etc. and this will be
> deployed in a full-fledged OS environment.
> ii. Why it is a reasonable overhead to have a full operating system
> rather than only a software package/framework? We do not have to take
> care about much with the operating system. 
> 
> C.) Decentralized Peer-to-Peer network with a two tier structure: 
> i. Masters are part of the p2p network slaves are not.
> ii. Slaves and Masters communicate over secured RPC calls.
> iii. There are two kinds of peers. Nodes and Supernodes. Nodes would
> only contribute to and profit from the central IP Data. Supernodes would
> also hold and distribute the distributed data(base). The following
> diagram gives an idea of the structure:
> http://ikinci.info/oisf-proposal.png   
> 
> Reasons for the P2P architecture:
> 1. Distributed database.
> 2. Nearly instant updates. P2P is the fastest way of distributing data.
> 3. P2P is the most hack-resistant architecture till date.
> 4. Independent from any vendor and thus always free signatures
> 
> Further advantages of this approach:
> 
> a) Scalability: from a home system to hundreds of systems connected to
> each other in a master slave configuration. Efficiently and consistently
> managing hundreds of distributed sensors worldwide! 
> 
> b) Virtualization and Containment Strategy: Are already built-ins from
> Ubuntu see JeOS Linux KVMs and App Armor.
> http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos
> http://www.ubuntu.com/products/whatisubuntu/serveredition/features/security
> Our system could be deployed on any PC with VMWare Player for example
> where we would only provide a ready to run full configured VMWare Image
> with the JeOS so that anyone could use our system with a couple of mouse
> clicks to protect only their windows pc connected to the net.
> 
> c) Support for many (scripting) languages (c, c++, fortran, perl,
> python, ruby, tcl etc.). We don't have to discourage capable developers/
> contributors only because we don't support their favorite programming
> language. And we can make use of the vast libraries and tools of these.
> Take a look at the many small scripts and tools in many different
> programming languages inside of BackTrack.
> 
> d) Built-in Security and software management with a big pool of actively
> maintained software. We don't have to have any software support for
> packages which are already part of the big Ubuntu repositories and only
> need to have updates and ports for our custom packages or special
> versions of our used tools. We therefore can use the development
> versions of the package maintainers etc. And we only had to provide an
> additional repository server with our packages on it.
> 
> e) Easy deployment, straightforward installation and out-of-the-box
> standard configuration. Maybe this is not for granny but her grandson
> could install this on an old spare pc or on a virtual machine on
> granny's laptop. Dual core laptops are already standard. But I think our
> system will be of most use for deployments who have at least a handful
> of PC's and a system admin.
> 
> f) Easy integration and administration of other packages for example
> Monitoring software like Nagios etc. 
> 
> g) Easy administration due to a standard web interface where features
> can be turned on and off and any further configuration would still be
> able over a shell and the applications configuration files. We could
> also support the installation of other maintenance tools like Webmin
> etc. so that the user gets powerful administration options and we don't
> have to write too much.
> 
> h) Easy additions to the standard distribution as the system is a
> standard Ubuntu/Debian installation.
> 
> i) Easy integration of honeypot/honeynet technologies: honeypots, client
> honeypots etc. for example nepenthes, honeyd etc.
> 
> j) Standardized release cycles of Ubuntu and guaranteed maintenance
> period.
> 
> k) passive OS fingerprinting (p0f and others)
> 
> We could start slow and implement features over time. The first version
> could be built in a couple of days! We could start with the approach of
> reusing software, and writing new software/glue code should only be the
> last resort.
> 
> Thanks for your attention and your precious time
> 
> Ali
> 
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion


-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------


From kevross33 at googlemail.com  Sat Feb 28 20:54:43 2009
From: kevross33 at googlemail.com (Kevin Ross)
Date: Sun, 1 Mar 2009 01:54:43 +0000
Subject: [Discussion] a few ideas
Message-ID: <e75cc74a0902281754h60e94c8fmb2e8e4cb60dba31f@mail.gmail.com>

Hey here are a few things I would like to see. Possibly already discussed or
not as useful as I think.

1) First Ossim (http://www.ossim.net/) has an interesting use of the
reliability of the sig, the priority of the host and some other things to
assign a risk to the attack. Using a similar system individual signatures
can be given a reliability which could mean sure fire attacks are flagged
immediately while unreliable signatures are not flagged immediately until
other factors are met. For instance under ossim you can basically say (in an
xml directive) if there are these snort sids it is a reliability of 3 and if
these snort sigs appear +2, if it persistant (for a set time) +1 and if a
web page error message appears +1 and so on. Using such a system could mean
false positives can automatically lowered while making more reliable attacks
against priority resources and the events related to that attack available
to the analyst (being able to define the priority of an asset such as a
server farm in comparison so the secretary's desktop would be useful). Also
things like if the attack was blocked by IPS or even a firewall if the logs
are available that the attack was mitigated the risk level can be reduced.

2) For active response the ability to specify agentless blocking devices.
i.e set up a pix/asa/iptables/pf firewall and say ssh into the device from
the master sensor, go into the correct configuration mode and then enter in
the appropriate command i.e ssh pix at firewall; enters the password; en;
enters the password; conf t; shun ip or acl. Then after a set time remove
the block. Also if it is an IPS the ability to say act inline but also have
active response such as dropping the attack and blocking the ip or an attack
is detected once drop it, if the same host attacks again block it
completely,

3) an installer on a cutdown linux/bsd system perhaps with a simple
installer, also perhaps configuration by a web interface. That way a
non-unix person can install the system selecting the relevant options, then
use the web interface to set up the distributed system. This would attract
more users by helping to simplify a basic setup. Possibly even installers
consisting of different tools, i.e an installer for master/slave sensors for
normal IDS/IPS and correlation and another say for a honeypot with nepenthes
or honeyd and in the install you can point it to the master sensor. That way
dedicated parts of the distributed system can be installed easily by
inexperienced users (which everyone will be who comes to use this system at
first till they learn it). Also using this methods means different types of
systems can be added to the distributed IDS/IPS as need dictates such as
some new type of detection tool to some future type of attack.

4) Perhaps the ability to either autofind or being able to enter in the
network topology it can determine the source of the attack within the
network kind of like csmars does (demos here
http://www.demolabs.co.uk/ciscoportal.htm). Also gathering information such
as hostname/netbios name, mac-address etc using tools like nbtscan on the
detection of a local attack (to avoid scanning outside the network which is
a bit scetchy). So if an attack is detected from an inside host (by
specifying rfc 1918 addresses) then execute information gathering tools to
provide more information to the analyst about the source or target of the
attack. That was it becomes easier to determine if it is an fp. i.e if there
is a buffer overflow for a windows system but the target determined by a
tool such as xprobe, nmap or whatever is some other OS and that information
is available immediately upon opening the even then the analyst has a better
understanding of the attack risk and likely result. Also such a system could
be intergrated into some sort of risk system, such as a netbios attack
against a linux system would lower the risk rating of the attack.

5) Perhaps the most ambitious of them all. If an attack is seen through
various methods, say a new worm. If it is unknown by the system and
confirmed by an analyst a signature can be created by the sensor, perhaps
with help from the analyst specifiying a few options like what to match upon
and distubuted to other systems around the world that choose to accept such
updates. Perhaps submitted and checked first by some central body to avoid
someone submitting fake sigs to the distributed system, then it can be
automatically downloaded by sensors which allow such updates. During a new
fast spreading worm this could mean sensors can be updated with this
information quickly with little intervention from the "clients" in the
distributed system such as homes and businesses.

I hope some of these ideas sound interesting.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20090301/aefe486b/attachment.html