[Discussion] Features - Designing for Scaleability

[Matt Jonkman]

John Pritchard wrote:
> I'd like to suggest that we take very large deployments into
> consideration when designing our solution. The kind of problems you
> encounter when managing an infrastructure with a handful or even a
> dozen different IDS sensors is very different than trying to
> efficiently and consistently manage infrastructures with larger
> footprints (e.g. > 100+ sensors).

Agreed, absolutely. I'm really envisioning a framework where we do away
with the idea of individual sensors. Get rid of the idea of a bunch of
independent points and more toward a meshed look. Where all sensors are
aware of each other and the data in each, maybe even the state of each.
How to do that I dunno. Maybe using the database as a communication medium?

> 
> A couple of suggestions to help our design address these potential
> deployment and management scenarios:
> 
> 1) Centralized sensor and policy management platform (or framework)
> --> Such a solution may be restricted to a single centralized server
> or multiple servers.
> --> Might be a parent -> child relationship among configuration
> servers to segregate business units, or simply replication among
> servers for disaster recovery / business continuity purposes

Ya, that's what I was going for above. I like the parent
child/department idea. Have a central db of course, and maybe
sub-sensors and sub parents. Then state and information can be shared
more readily among related senors, like by location, or by perimeter vs
internal, user nets vs servers, etc.


> --> efficient, repeatable, and audit-able methodology for making
> changes to both individual sensors as well as pre-defined groups of
> sensors (e.g. dmz sensors, dns sensors, development lab sensors,
> etc...)

I want a db based config for all sensors. The whole config file approach
to it is a pain. It might be easier to run a db loaded config and
provide a basic web page or config tool to load these into the db. Then
it'd be pretty easy to maintain change tracking and give permissions for
changes, etc.

> --> My experience to date has been performing these kind of tasks with
> home-grown scripts, ssh, scp, audit logs, etc... However, it would be
> nice to find something a little more mature for this project.
> 
> I have not used it, but there is a project called "Puppet" that looks
> like it might be a good candidate for trying to develop a framework
> along these lines:
> http://reductivelabs.com/trac/puppet/wiki/PuppetIntroduction
> 

Very interesting, anyone have experience with it?

> 
> 2) Centralized sensor and policy monitoring platform (or framework)
> --> This is similar to the "management framework" concept, but the
> focus is more on device health monitoring... and possibly policy
> integrity monitoring...
> --> For this piece, I'm thinking of something that provides functions
> such as looking at memory utilization, cpu utilization, hard-drive
> space, network interface stats... and other "bits" such as dates and
> checksums for critical configuration files changed (e.g. detection
> policies, tuning policies, variable definitions, etc)...
> 

Absolutely, great idea! I think that could even be db based as well.
insert stats every x seconds. Makes the sensors more disposable as well
as all data is in the db. Just plug in a new box and tell it where to
get/store it's stuff via db.


> 
> 3) Distributed Data Repositories
> I know we briefly touched on the database stuff when talking about
> schema design. I just wanted to add a plug for a couple of things
> here:
> --> encrypted communication channels (sensor -> DB or sensor -> sensor)
> --> ability to simultaneously log to 2 or more data repositories

Definitely on both. We know we'll need to support postgress, mysql and
oracle. Anything else?

I believe all three have an ssl or otherwise based encryption option.
Should be look to make our own vpn or rely on these?

Matt

> 
> I strongly agree with the concept of designing modular solutions. So,
> these kind of features can be "bolted on" if they were required.
> 
> Look forward to everyone's thoughts on how we can most effectively
> tackle problems of scale for large deployments.
> 
> Cheers, John
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion

-- 
--------------------------------------------
Matthew Jonkman
Emerging Threats
Phone 765-429-0398
Fax 312-264-0205
http://www.emergingthreats.net
--------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc