[Discussion] The Revealer a proposal for the solution to many of our problems

Sat Feb 28 14:34:41 UTC 2009

Hi Ali,

In general I'd like to say: very interesting ideas, but I think most are
outside of the scope of this project. What we're building here is an
actual detection engine.

Further comments inline:

Ali Ikinci wrote:
<snip>
> I am thinking of a way how we can bundle all ideas, avoid duplicate
> work, contribute to the community, build a (the) big IP/DNS reputation
> system, invent/start new software projects/frameworks, make use of all
> available systems, keep it simple for us and the users and be not
> dependent on a vendor or authority..

The independence is the reason we're organized as a non-profit.

> So here are some trade-offs which I think we have to deal with:
> 
> Not 'reinventing the Wheel' vs. Developing something new
> There are already many good software packages/frameworks who are very
> good at a particular matter and already have a great pool of reputation
> data. Why not use all of them (maybe only for one purpose) than we get a
> system which is good at all minor aspects and features. We can at
> anytime contribute to the projects or start our own sub projects with
> features and functionality we want to add.
> 
> A quote from John Pritchard's post:
> "I'd like to suggest that we take very large deployments into
> consideration when designing our solution. The kind of problems you
> encounter when managing an infrastructure with a handful or even a
> dozen different IDS sensors is very different than trying to
> efficiently and consistently manage infrastructures with larger
> footprints (e.g. > 100+ sensors)."
> 
> Scalability and many features vs. Complexity for granny and her grandson
> When we need to deploy a system with so many features and such a
> complexity which involves the latest threats and detection techniques
> the vast majority of people including many well read sysadmins will have
> it hard to implement and configure all features of this system and even
> harder to scale this to more than one system. Whereas we need to have as
> much users as we can get to have more sensors out there waiting and
> delivering reputation data to us and thus to the whole community.

I think our engine should work as much as possible "out of box".
Additionally I think it should be suitable for being configured by
external programs like frontends etc.

> Centralization vs. Distribution
> Do we need a central instance to store data? What if that instance gets
> bought up by a company or gets otherwise compromised.

I don't think OISF is actually going to maintain the reputation lists,
that is maybe more something for Emerging Threats and other 3rd parties.
For legal reasons too, since we don't want to be able to be sued for
listing someone and risking losing the development funds.

I think our engine should be able to use an existing reputation data
format standard, or develop one.

> A New Signature/Policy format vs. various available signature/policy
> formats:
> There are already a wide variety of tools and formats. If we use only
> one of these tools then we will need to adopt their format and will also
> inherit all their shortcomings. Like in this discussion has been about
> snort and bro. Whereas if we use multiple programs we need to administer
> multiple formats. If we build a new format than we can not profit from
> the reputation data which is already there or we have to convert in some
> or the other way.

My personal preference would be to use the Snort format as many people
out there already know it. We may have to extend it in some ways, making
it non standard.

> Forking projects vs. Rewriting similar projects vs. Extension of
> available projects: 
> Forking is bad and only a last solution to an otherwise unresolvable
> problem. Free projects normally remain to be free for their whole life
> so if we want to have new features we should try and add them to the
> existing projects. Only customizations to our system can be written as
> sub projects.

The engine we're building is going to be a from scratch implementation.

"The Revealer" proposal below I think falls outside of our scope. It
does sound like a great idea for a project though!

Cheers,
Victor

> Proposal: "The Revealer"
> 
> Contrary to a software package/framework as the outcome of this
> discussion: A custom full-fledged Ubuntu-Server edition customized to
> and configured from the start to be part of a master-slave P2P network
> of OISF nodes called 'the Revealer' (or Seeker) (One could take ideas
> out of systems like Roo but also IPCop,m0n0wall etc.):
> 
> Featuring many available solutions for various detection/categorization
> fields bundeled with monitoring and management interfaces.
> 
> This system could operate as a Layer2 bridge(or a network tap? or as a
> router?).
> 
> Next Gen ideas and technologies should be built-in to be long lived,
> usable and up-to date like Virtualization, P2P, Web 2.0, Honeypots/nets,
> Cloud Computing etc.
> 
> We would then add sub projects(for example a new ids system) which might
> also support plug-ins. These should be self-contained free-software
> projects so that they can be reused in other projects/deployments.
> 
> The Revealer distribution should be modular and modules (like
> snort,bro,our ids etc.) maybe de/activated during runtime with a click
> on the web interface. These modules would come with a good standard
> configuration and submission setting.
> 
> There would be one central Monitoring and Management web interface which
> would be located at the master server. Including centralized patch and
> signature management. And many slaves connected and would be
> administered from the master.
> 
> A new meta policy format which would become a sort of a database of what
> is already there for bro and snort and which would also contain its own
> information regarding the signatures. For example to have a snort
> signature together with the human readable information about who has
> written it and how it was detected etc. This new policy format should
> also be used as the main policy format for other detection (sub)projects
> which we write ourselves. Think of the CVE database.
> 
> Finally we would get the best database of threats and contribute heavily
> to the security of the net as a whole. Global culprits would be
> identified and banned accurately and without a payment/subscription to a
> third party!
> 
> Further ideas regarding the Revealer:
> 
> I.) Redirecting crackers to server honeypots: We could also
> automatically redirect attackers trying to crack into remote peers to
> our local honeypots or other researchers honeynets and thus collect all
> crackers/malware at well observed and researched honeynets. This could
> be done using deception techniques to trick the attacker to believe they
> are still attacking the same ip so all traffic from the attacker should
> be routed over the initially attacked node.
> 
> II.) Redirecting malicious traffic to client honeypots: We could
> built-in a mechanism to gather all kind of interesting file types out of
> a network flow (this has already been suggested but we need client
> honeypot detection mechanisms in addition to AV signatures) like
> executables, JavaScript, ActiveX, VBS, PDF, SWF, OLE etc. which could be
> malicious to the requesting clients and hand them over to various
> AV-Solutions,client honeypots (like Caffeine Monkey, phoneyc, honeyc
> etc.) and CWSandbox. Some of these could directly run on the Revealer
> itself.
> 
> III.) A simple user contribution interface at the main web interface of
> each master node. This would enable the user to submit various
> threats/malicious files/suspicious things etc. So she could use this
> simple interface to anonymously submit suspicious files to us (and/or to
> CWSandbox, Virustotal) or URL's or comments about suspicious behavior
> etc. This is something which might become a powerful feature. Maybe also
> a place to submit self written snort/bro signatures and the like. A
> widespread and active community means power.
> 
> IV.) Built-in support for KVM based Ubuntu JeOS installations for
> example as a containment strategy for honeypots etc. 
> 
> Some reasons for the design decisions:
> 
> A.) Automated deployment, configure once deploy 100x times.
> http://www.ubuntu.com/products/whatisubuntu/serveredition/features/autodeploy
> 
> B.) Full-fledged OS vs. A software package/framework:
> i. The question is not an either/or but an also. We are going to have a
> software package/framework with many sub projects etc. and this will be
> deployed in a full-fledged OS environment.
> ii. Why it is a reasonable overhead to have a full operating system
> rather than only a software package/framework? We do not have to take
> care about much with the operating system. 
> 
> C.) Decentralized Peer-to-Peer network with a two tier structure: 
> i. Masters are part of the p2p network slaves are not.
> ii. Slaves and Masters communicate over secured RPC calls.
> iii. There are two kinds of peers. Nodes and Supernodes. Nodes would
> only contribute to and profit from the central IP Data. Supernodes would
> also hold and distribute the distributed data(base). The following
> diagram gives an idea of the structure:
> http://ikinci.info/oisf-proposal.png   
> 
> Reasons for the P2P architecture:
> 1. Distributed database.
> 2. Nearly instant updates. P2P is the fastest way of distributing data.
> 3. P2P is the most hack-resistant architecture till date.
> 4. Independent from any vendor and thus always free signatures
> 
> Further advantages of this approach:
> 
> a) Scalability: from a home system to hundreds of systems connected to
> each other in a master slave configuration. Efficiently and consistently
> managing hundreds of distributed sensors worldwide! 
> 
> b) Virtualization and Containment Strategy: Are already built-ins from
> Ubuntu see JeOS Linux KVMs and App Armor.
> http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos
> http://www.ubuntu.com/products/whatisubuntu/serveredition/features/security
> Our system could be deployed on any PC with VMWare Player for example
> where we would only provide a ready to run full configured VMWare Image
> with the JeOS so that anyone could use our system with a couple of mouse
> clicks to protect only their windows pc connected to the net.
> 
> c) Support for many (scripting) languages (c, c++, fortran, perl,
> python, ruby, tcl etc.). We don't have to discourage capable developers/
> contributors only because we don't support their favorite programming
> language. And we can make use of the vast libraries and tools of these.
> Take a look at the many small scripts and tools in many different
> programming languages inside of BackTrack.
> 
> d) Built-in Security and software management with a big pool of actively
> maintained software. We don't have to have any software support for
> packages which are already part of the big Ubuntu repositories and only
> need to have updates and ports for our custom packages or special
> versions of our used tools. We therefore can use the development
> versions of the package maintainers etc. And we only had to provide an
> additional repository server with our packages on it.
> 
> e) Easy deployment, straightforward installation and out-of-the-box
> standard configuration. Maybe this is not for granny but her grandson
> could install this on an old spare pc or on a virtual machine on
> granny's laptop. Dual core laptops are already standard. But I think our
> system will be of most use for deployments who have at least a handful
> of PC's and a system admin.
> 
> f) Easy integration and administration of other packages for example
> Monitoring software like Nagios etc. 
> 
> g) Easy administration due to a standard web interface where features
> can be turned on and off and any further configuration would still be
> able over a shell and the applications configuration files. We could
> also support the installation of other maintenance tools like Webmin
> etc. so that the user gets powerful administration options and we don't
> have to write too much.
> 
> h) Easy additions to the standard distribution as the system is a
> standard Ubuntu/Debian installation.
> 
> i) Easy integration of honeypot/honeynet technologies: honeypots, client
> honeypots etc. for example nepenthes, honeyd etc.
> 
> j) Standardized release cycles of Ubuntu and guaranteed maintenance
> period.
> 
> k) passive OS fingerprinting (p0f and others)
> 
> We could start slow and implement features over time. The first version
> could be built in a couple of days! We could start with the approach of
> reusing software, and writing new software/glue code should only be the
> last resort.
> 
> Thanks for your attention and your precious time
> 
> Ali
> 
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------