[Discussion] I think everyone is describing Bro

[Seth Hall]

Sorry, I just joined the list so I'm going to be doing some odd  
quoting from the list archive :)  I do want to point out too, that I'm  
not writing this email to downplay OISFs goals but rather to hopefully  
guide OISF toward improving an existing opensource project (Bro - http://www.bro-ids.org/ 
  ) that already does much of what is being discussed on this list.

Andre Ludwig wrote:
 > 1.  Signature matching has its use but can not be an end all for  
detection of attacks.

Completely agreed and to be honest, at OSU (The Ohio State University)  
we really don't run many signatures with Bro.  The only signature that  
we get any real use out of is one that matches remote Windows shells  
on arbitrary ports.  We found that the scripts Bro ships with and the  
scripts that I was writing were detecting all of the activity that we  
were interested in.  It has continued that it's usually easy to write  
scripts to detect new activity that we might be interested in.

 > 2.  Performance and wire speed is always going to be an issue
<snip>

Agreed.  The main Bro developers are already working on several  
techniques to scale Bro's performance.  There is the cluster  
deployment which already works (http://www.icir.org/robin/bro-cluster/README.html 
).  We (OSU) run a cluster deployment of 21 worker nodes currently and  
tests have been done that show that it scales linearly to 24 nodes.

Bro has a communications framework so that the cluster workers can  
communicate with each other too.  This is useful for the times when  
you need some information shared around to all of the workers (because  
they each only see their portion of all traffic) like when one host  
identifies a host that is SSH scanning/bruteforcing, it can let all of  
the other nodes know about that ip address automatically.  The best  
part is that it's all scalable with commodity hardware, there isn't  
any non-commodity hardware in our entire cluster. :)

The Bro developers have also begun to parallelize their core engine  
and once that is working (not yet) Bro users will see even greater  
performance increases if they're running multi core/processor machines.

David Glosser wrote:
 > ways to get alerts for fast-flux and other things where current  
sigs are prone to FPs

I implemented the algorithm behind the Holz et al paper in just a day  
or two after the paper was released.  It was all done in the Bro  
scripting language.  To be fair, the guy   that did the snort  
preprocessor (Scott Campbell) for detecting fastflux also wrote a Bro  
script for detecting fastflux at the same time as me. :)

Matt Jonkman wrote:
 > We can't give huge lists of bad IPs to most tools, we can't feed  
behavior patterns to existing tools

Bro is a perfect fit for this use case.  It can take fairly gigantic  
lists of IP addresses and check for membership in the lists very  
frequently without suffering performance-wise.

 > 1. Native multithreading.

Like I wrote earlier, that's being worked on now. :)

 > 2. IP Reputation Sharing

It's only waiting for someone to extend the Bro communications  
framework to do inter-site communications.  There isn't currently any  
way to specify what you want to receive from remote sites or how much  
you trust them.  It would be great project for someone to pick up.

 > 3. Native ipv6

Done.  Needs more real world testing though, since high volume real  
world testing with IPv6 is a little difficult to come by still.

 > 5. Scoring

I can definitely imagine implementing something like this with Bro,  
but you might realize that this isn't quite as important as you  
thought once you move away from the IDS==signatures mindset.

Jeremy wrote:
 > Network intelligence through passive identification.
<snip>
 > a windows xp home
 > computer correlated with network flow data and port data could easily
 > be alerted on for any outgoing port 25 connections

That's a great idea Jeremy!  There's no reason it couldn't be  
implemented in Bro too.  I might do that soon, it would be useful for  
us.

Andre Ludwig wrote:
 >7. Ability to look across packets in a stream, be it 3 packets  
before or 4 packets ahead.

Bro doesn't tend to work at the individual packet level, so this is  
not really an issue.

 > 8.  Some sort of meta language needs to be created that easily and  
effectively can communicate any applications "state".

If I'm understanding correctly, you want a language that allows you to  
program your analysis?  Here are the docs for the Bro policy language:
    http://www.bro-ids.org/wiki/index.php/Reference_Manual

John Ives wrote:
 > 9. The ability to pull files out of the stream in real-time.

There is a new script in Bro 1.4 that was just released on Friday.   
This is from the changelog...
     - The new script http-extract-items.bro extracts the items from  
HTTP
        traffic into individual files (Vern Paxson).

At OSU, we have also been running a script I wrote called http- 
identified-files.bro (included with Bro 1.4) that identifies file  
types on all http transfers with libmagic.  The result is that we have  
a log of every Windows and UNIX executable that is downloaded in or  
out of our network over HTTP (on any port and regardless of URL).  Bro  
can identify any file type that the libmagic database you are using  
can identify.

 > a language/protocol for it to interact with other security devices

Bro already has this, it's called Broccoli and it's included with the  
main Bro distribution.  Here are the docs:
   http://www.bro-ids.org/broccoli/index.html

As some examples of what the library can do, I'm using it in two ways  
currently.
1. I'm pushing authentication data from a number of different OSU  
services into our Bro cluster so that it can be analyzed along with  
the network traffic.  We get the benefit of continuing to do our  
anomaly detection in a single language that is very well suited for  
detecting odd activity over time.
2. I just released a tool that connects directly to my master node of  
my cluster and logs data from the cluster directly to a postgresql  
database in realtime.  There is no hacky file tailing being done.  Bro  
sends an event to my tool it logs the data in the message to the  
database.  Here's the bro-dblogger:
     http://github.com/sethhall/bro-dblogger/tree/master

There are also currently functioning bindings for python and ruby.   
Python bindings are included in the Bro distribution now I believe and  
the Ruby bindings can be installed with the "broccoli" gem.

Victor Julien wrote:
 > For example in ModSecurity you can match on the password on HTTP  
basic
 > auth by capturing the string containing the username and password  
from
 > the HTTP header, splitting it, decoding it's base64

This is absolutely possible today with Bro.  If you're interested in  
it for some reason, I could write the script to show how straight  
forward it is.

Whew, sorry about the long email but I hadn't seen a reference to Bro  
in the archives yet and there are so many suggestions that are already  
possible in Bro or would be easy to implement that I couldn't pass it  
up.  I'd certainly be willing to answer any questions people have  
about Bro.  It's getting late here and I might not make much sense in  
part of my email. :)

  .Seth

---
Seth Hall
Network Security - Office of the CIO
The Ohio State University
Phone: 614-292-9721