[Discussion] The Revealer a proposal for the solution to many of our problems

[Ali Ikinci]

Before I present my proposal I want to say that I am a relatively new
security researcher and a software engineer. I am working on my own
client honeypot (http://monkeyspider.sf.net) and am not so well read on
IDS/IPS. I am writing this down in a kind of brainstorming session of
mine over a few days and while reading the list archive and taking
notes, so don't take anything here too serious. It might contain a dozen
of mistakes and might not be well formed.

In this post I do not want to go into too much detail about detection
mechanisms and features of our system as there are already many great
ideas from real pros in this list. There is no doubt that these ideas
should all be cherished in some or the other way. While doing this I
think we have the problem of not seeing the forest for the trees. We
know what we need to do and how we can detect threats and collect IP/DNS
reputation data but not how we can find a home for all this ideas while
keeping it simple to assure a wide use and profit from many sensors
worldwide. That's where this proposal comes in.

I am thinking of a way how we can bundle all ideas, avoid duplicate
work, contribute to the community, build a (the) big IP/DNS reputation
system, invent/start new software projects/frameworks, make use of all
available systems, keep it simple for us and the users and be not
dependent on a vendor or authority..

So here are some trade-offs which I think we have to deal with:

Not 'reinventing the Wheel' vs. Developing something new
There are already many good software packages/frameworks who are very
good at a particular matter and already have a great pool of reputation
data. Why not use all of them (maybe only for one purpose) than we get a
system which is good at all minor aspects and features. We can at
anytime contribute to the projects or start our own sub projects with
features and functionality we want to add.

A quote from John Pritchard's post:
"I'd like to suggest that we take very large deployments into
consideration when designing our solution. The kind of problems you
encounter when managing an infrastructure with a handful or even a
dozen different IDS sensors is very different than trying to
efficiently and consistently manage infrastructures with larger
footprints (e.g. > 100+ sensors)."

Scalability and many features vs. Complexity for granny and her grandson
When we need to deploy a system with so many features and such a
complexity which involves the latest threats and detection techniques
the vast majority of people including many well read sysadmins will have
it hard to implement and configure all features of this system and even
harder to scale this to more than one system. Whereas we need to have as
much users as we can get to have more sensors out there waiting and
delivering reputation data to us and thus to the whole community.

Centralization vs. Distribution
Do we need a central instance to store data? What if that instance gets
bought up by a company or gets otherwise compromised.

A New Signature/Policy format vs. various available signature/policy
formats:
There are already a wide variety of tools and formats. If we use only
one of these tools then we will need to adopt their format and will also
inherit all their shortcomings. Like in this discussion has been about
snort and bro. Whereas if we use multiple programs we need to administer
multiple formats. If we build a new format than we can not profit from
the reputation data which is already there or we have to convert in some
or the other way.

Forking projects vs. Rewriting similar projects vs. Extension of
available projects: 
Forking is bad and only a last solution to an otherwise unresolvable
problem. Free projects normally remain to be free for their whole life
so if we want to have new features we should try and add them to the
existing projects. Only customizations to our system can be written as
sub projects.

Proposal: "The Revealer"

Contrary to a software package/framework as the outcome of this
discussion: A custom full-fledged Ubuntu-Server edition customized to
and configured from the start to be part of a master-slave P2P network
of OISF nodes called 'the Revealer' (or Seeker) (One could take ideas
out of systems like Roo but also IPCop,m0n0wall etc.):

Featuring many available solutions for various detection/categorization
fields bundeled with monitoring and management interfaces.

This system could operate as a Layer2 bridge(or a network tap? or as a
router?).

Next Gen ideas and technologies should be built-in to be long lived,
usable and up-to date like Virtualization, P2P, Web 2.0, Honeypots/nets,
Cloud Computing etc.

We would then add sub projects(for example a new ids system) which might
also support plug-ins. These should be self-contained free-software
projects so that they can be reused in other projects/deployments.

The Revealer distribution should be modular and modules (like
snort,bro,our ids etc.) maybe de/activated during runtime with a click
on the web interface. These modules would come with a good standard
configuration and submission setting.

There would be one central Monitoring and Management web interface which
would be located at the master server. Including centralized patch and
signature management. And many slaves connected and would be
administered from the master.

A new meta policy format which would become a sort of a database of what
is already there for bro and snort and which would also contain its own
information regarding the signatures. For example to have a snort
signature together with the human readable information about who has
written it and how it was detected etc. This new policy format should
also be used as the main policy format for other detection (sub)projects
which we write ourselves. Think of the CVE database.

Finally we would get the best database of threats and contribute heavily
to the security of the net as a whole. Global culprits would be
identified and banned accurately and without a payment/subscription to a
third party!

Further ideas regarding the Revealer:

I.) Redirecting crackers to server honeypots: We could also
automatically redirect attackers trying to crack into remote peers to
our local honeypots or other researchers honeynets and thus collect all
crackers/malware at well observed and researched honeynets. This could
be done using deception techniques to trick the attacker to believe they
are still attacking the same ip so all traffic from the attacker should
be routed over the initially attacked node.

II.) Redirecting malicious traffic to client honeypots: We could
built-in a mechanism to gather all kind of interesting file types out of
a network flow (this has already been suggested but we need client
honeypot detection mechanisms in addition to AV signatures) like
executables, JavaScript, ActiveX, VBS, PDF, SWF, OLE etc. which could be
malicious to the requesting clients and hand them over to various
AV-Solutions,client honeypots (like Caffeine Monkey, phoneyc, honeyc
etc.) and CWSandbox. Some of these could directly run on the Revealer
itself.

III.) A simple user contribution interface at the main web interface of
each master node. This would enable the user to submit various
threats/malicious files/suspicious things etc. So she could use this
simple interface to anonymously submit suspicious files to us (and/or to
CWSandbox, Virustotal) or URL's or comments about suspicious behavior
etc. This is something which might become a powerful feature. Maybe also
a place to submit self written snort/bro signatures and the like. A
widespread and active community means power.

IV.) Built-in support for KVM based Ubuntu JeOS installations for
example as a containment strategy for honeypots etc. 

Some reasons for the design decisions:

A.) Automated deployment, configure once deploy 100x times.
http://www.ubuntu.com/products/whatisubuntu/serveredition/features/autodeploy

B.) Full-fledged OS vs. A software package/framework:
i. The question is not an either/or but an also. We are going to have a
software package/framework with many sub projects etc. and this will be
deployed in a full-fledged OS environment.
ii. Why it is a reasonable overhead to have a full operating system
rather than only a software package/framework? We do not have to take
care about much with the operating system. 

C.) Decentralized Peer-to-Peer network with a two tier structure: 
i. Masters are part of the p2p network slaves are not.
ii. Slaves and Masters communicate over secured RPC calls.
iii. There are two kinds of peers. Nodes and Supernodes. Nodes would
only contribute to and profit from the central IP Data. Supernodes would
also hold and distribute the distributed data(base). The following
diagram gives an idea of the structure:
http://ikinci.info/oisf-proposal.png   

Reasons for the P2P architecture:
1. Distributed database.
2. Nearly instant updates. P2P is the fastest way of distributing data.
3. P2P is the most hack-resistant architecture till date.
4. Independent from any vendor and thus always free signatures

Further advantages of this approach:

a) Scalability: from a home system to hundreds of systems connected to
each other in a master slave configuration. Efficiently and consistently
managing hundreds of distributed sensors worldwide! 

b) Virtualization and Containment Strategy: Are already built-ins from
Ubuntu see JeOS Linux KVMs and App Armor.
http://www.ubuntu.com/products/whatisubuntu/serveredition/jeos
http://www.ubuntu.com/products/whatisubuntu/serveredition/features/security
Our system could be deployed on any PC with VMWare Player for example
where we would only provide a ready to run full configured VMWare Image
with the JeOS so that anyone could use our system with a couple of mouse
clicks to protect only their windows pc connected to the net.

c) Support for many (scripting) languages (c, c++, fortran, perl,
python, ruby, tcl etc.). We don't have to discourage capable developers/
contributors only because we don't support their favorite programming
language. And we can make use of the vast libraries and tools of these.
Take a look at the many small scripts and tools in many different
programming languages inside of BackTrack.

d) Built-in Security and software management with a big pool of actively
maintained software. We don't have to have any software support for
packages which are already part of the big Ubuntu repositories and only
need to have updates and ports for our custom packages or special
versions of our used tools. We therefore can use the development
versions of the package maintainers etc. And we only had to provide an
additional repository server with our packages on it.

e) Easy deployment, straightforward installation and out-of-the-box
standard configuration. Maybe this is not for granny but her grandson
could install this on an old spare pc or on a virtual machine on
granny's laptop. Dual core laptops are already standard. But I think our
system will be of most use for deployments who have at least a handful
of PC's and a system admin.

f) Easy integration and administration of other packages for example
Monitoring software like Nagios etc. 

g) Easy administration due to a standard web interface where features
can be turned on and off and any further configuration would still be
able over a shell and the applications configuration files. We could
also support the installation of other maintenance tools like Webmin
etc. so that the user gets powerful administration options and we don't
have to write too much.

h) Easy additions to the standard distribution as the system is a
standard Ubuntu/Debian installation.

i) Easy integration of honeypot/honeynet technologies: honeypots, client
honeypots etc. for example nepenthes, honeyd etc.

j) Standardized release cycles of Ubuntu and guaranteed maintenance
period.

k) passive OS fingerprinting (p0f and others)

We could start slow and implement features over time. The first version
could be built in a couple of days! We could start with the approach of
reusing software, and writing new software/glue code should only be the
last resort.

Thanks for your attention and your precious time

Ali