[Discussion] Virtues of a Multi-Sensor Environment

[Martin Holste]

I find both Sguil and Bro to be venerable platforms, however Seth, I think
that Bro is better left as a sniffer/preprocessor.  I think that the bulk of
the processing (I would even argue things like the libmagic part) are better
left to another process that Bro ships interesting things to via Broccoli.
While Bro sports a lot of really good plugins and a fair amount of
extensibility, I think it's the wrong design path to throw everything into
it as a framework.  It is no accident that it lacks native database support
and has no frontend, because it wasn't designed to be all-encompassing, it
was designed to do the heavy lifting and pass the rest off via Broccoli.
But Bro does have the great strength of allowing for extremely rich
signature scripts, as you pointed out.

David, (I am a regular reader of your blog, BTW):  I consider Sguil the
closest thing to an out-of-the-box solution for NSM out there, and I think
that it is 100% on the right track.  In my opinion, though, the entire Sguil
architecture feels archaic and a bit kludgy, and I definitely require a web
console as opposed to a Tcl-based console.  I completely understand why Tcl
was the original design choice, but AJAX has really made it obsolete.  My
vote would be to extend the NSM ideas of Sguil to a platform based on a
language with a wider audience, like Perl.  Admittedly, since I know Perl
the best, I will of course advocate for it (just as you advocated Sguil and
Seth advocated Bro).  That said, you can take a Perl class (or Java, .NET,
or probably even Python) at any tech college, but very few places will teach
Tcl.  Perl also has a much richer library of modules to draw on to do all of
the really interesting things that you talked about in your last post.  As
an example, as of this morning, all packed files downloaded in my org are
extracted from the network and sent to VirusTotal, Symantec, and now
CWSandbox, and the results are collected and stored in a local database for
data mining purposes.

In any case, my goal is not to start a flame war over which programming
language is best, but many vendors (like VMWare, HP, IBM, Cisco) chose to
use Perl as the language in which to publish modules because of its ubiquity
and extensibility.  I think that most of our work will be analytical in
nature, and this work is best done away from the packet sniffing/recording
environment in a framework based on a dynamic language.  I furthermore think
that the best-suited dynamic language is Perl (maybe even Perl 6's Parrot
platform).  All of this is of course debatable, and I welcome rebuttals.

--Martin

On Wed, Oct 29, 2008 at 2:55 PM, Seth Hall <hall.692 at osu.edu> wrote:

>
> On Oct 29, 2008, at 2:13 PM, David J. Bianco wrote:
>
> > There was some discussion on this list about Bro.  I'm a big fan of
> > Bro.
> > It performs very well in the pattern analysis and policy-based
> > detection
> > arenas, in which Snort (and thus, Sguil) do poorly.  Conversely, Bro's
> > signature matching abilities aren't very good, and it certainly
> > doesn't have
> > a robust sig library like Snort does.
>
> Bro's signature matching is only poor because it isn't used much by
> anyone that uses Bro heavily.  I don't believe that it would be too
> much of a project to flesh out the signature matching capabilities of
> Bro to mirror Snort.  One area where Bro's signature matching already
> bests Snort is in its ability to have bidirectional signatures.  You
> can define a signature that only triggers if another signature is seen
> in the opposite direction of the connection.  For example, if you're
> matching something that requests a certain URL, maybe you only care
> about it the server responds to the client with a 200 status code.
> This is easy to do with bidirectional signatures.  You can also have
> signature matches thrown into Bro policy script code so that you can
> do any number of extended heuristics based on other information Bro
> has detected.
>
> > My view is that they are both complementary technologies, and should
> > be
> > deployed in tandem.  It has been suggested on this list that Bro be
> > the
> > engine for future OISF implementation.  Great idea!  But I propose
> > that
> > we consider our project a framework, or suite of tools, and that Bro
> > be
> > a critical *piece* of that framework, but  not the only framework.
>
> In my opinion, Bro would actually make a very general framework to
> build upon.  Bro is already more or less a general network analysis
> framework.  It has support for lots of various technologies that
> people want in order to do network traffic analysis and typically it's
> fairly easy to add new functionality building on top of existing
> functionality.  As an example of this, yesterday I wrote some small
> extensions that will identify windows executables being transferred
> over HTTP with libmagic and once a windows executable is identified it
> will begin building an MD5 hash of the data.  It then takes the MD5
> hash and checks it with the newly announced Team Cymru Malware Hash
> Repository[1] in realtime to see if they have that binary file in
> their malware repository.  If you would like to see how little code I
> had to write to make it work, here's the link to my changes:
>   http://github.com/sethhall/bro_scripts/tree/master/md5_hash_malware
>
> Malware Hash Registry docs:
>   http://www.team-cymru.org/Services/MHR/
>
> My point is just that Bro is very much in a position to be a central
> framework.  Much more so than any other tools currently available.
>
> > I can envision a scenario where I'd deploy, for example, signature
> > based
> > IDS (Snort), policy based IDS (Bro), correlated event alerts (OSSEC)
> > and
> > perhaps even application specific sensors (mod_security, database IDS,
> > spear phishing detection, etc).
>
> The problem is that you're replicating effort there.  You'd have to
> maintain multiple machines just to run different tools (assuming
> you're looking at too much traffic to analyze on a single machine with
> multiple tools).  As I mentioned above, the problems with Bro's
> signature support are not due to a major failing in the software it's
> just temporarily suffering due to a lack of interest from the Bro
> community. :)
>
> It probably makes sense to feed the data from other tools like OSSEC
> into Bro first too because it has very good support for writing
> scripts that deal with data expiration over time and catching time
> based events.  We use it to look for anomalies and normalize our
> authentication logs by pushing a lot of them into Bro with the
> Broccoli library (it's included with Bro).  I know of another location
> that was using instrumented sshd servers to forward the content of all
> shell sessions to their Bro instance and looking for anomalies with
> the commands being typed over ssh to catch compromised accounts.
>
> After Bro has a chance to look at the data, it can then pass
> normalized and aggregated data (and maybe more notices if it caught
> anything strange in the data) into Squil and its database.
>
> > All these events should go to the same
> > analysis tool, they should be organized effectively, searchable,
> > cross-referenced, and tied to forensic data (network sessions, packet
> > captures, log files, etc).
>
> I definitely agree with this!  Bro doesn't have anything yet that is
> end user focused towards doing forensics with the data being
> generated.  If Squil could give it that frontend, that would be awesome.
>
> > In fact, we're much of the way there already.  I released an agent to
> > slurp OSSEC events into Sguil quite some time ago, and we have done
> > the
> > same with other data sources as well.  Communication between Bro and
> > Sguil
> > is on my TODO list as well.  So really, the idea of deploying multiple
> > types of sensors feeding into the same analysis tool is a practical
> > one.
> > I think this should be a design goal of the project.
>
> If you'd like assistance with integrating Squil and Bro, I'd be glad
> to give some pointers or lend a hand with it. :)
>
>   .Seth
>
> ---
> Seth Hall
> Network Security - Office of the CIO
> The Ohio State University
> Phone: 614-292-9721
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081029/f0c0b223/attachment-0002.html>