[Discussion] Features

[Jeremy]

Comments are inline but first I would like to add an item:

6.  Network intelligence through passive identification.  What I mean
by this is like Snorts RNA, NMAP, and p0s fingerprinting to
dynamically update inventory data for correlation engine usage.  I
think we all know how great it would be to have this data which could
dynamically or auto magically configure/reconfigure signatures and
correlation engines to align with applications and operating systems
to truly substantiate the risk of attacks and/or threats.  Having the
availability of correlated network intelligence could make tremendous
headway in the world of anomaly detection i.e.  a windows xp home
computer correlated with network flow data and port data could easily
be alerted on for any outgoing port 25 connections, as they are most
likely the newest member of a spam bot network.  Obviously this is
just a trivial example but it still is not a trivial task to pull off
even with the most advanced SIM.  This would also allow for
operational performance enhancements by providing context regarding
the network they operate in.  The dynamic configuration and
reconfiguration of network devices to ignore threats that are not
applicable would definitely enhance performance by reducing overhead
caused by signatures and other security measures.

On Thu, Oct 16, 2008 at 8:00 PM, Matt Jonkman <jonkman at jonkmans.com> wrote:
> Here's the big thread. And don't be afraid to start sub-threads for
> specifics here.
>
> The features we want to go after here are the primary reason we sought
> this funding and are taking this challenge on. Existing stuff works, but
> there's SO much more we could be doing by looking past traditional ips
> strengths. The challenge is that those things aren't conducive to making
> a commercial product with millions invested in development. No one can
> take this risk now, so we're going this route to make it happen.
>
> We have information about bad guys, bad places, and bad patterns. Lots
> of it, terabytes of it. We've got gigs of data about bad stuff in the
> sandnet at emerging threats alone. But most of that we can't effectively
> act upon. We can't give huge lists of bad IPs to most tools, we can't
> feed behavior patterns to existing tools, we can't share scan data
> globally, etc.
>
> So here we are. I have things I wish I could do, you have things you
> wish you could do, over the next couple of months we aim to get to the
> core set of the most important things that most of us want to be able to
> do. Then we'll go after it.
>
> So here's my wish list:
>
> 1. Native multithreading.
> Not each preprocessor or post processor can go to a thread, but each
> stream can take a thread. Think apache. More servers = more requests
> served. THe complications of sharing state between them and the like is
> a challenge, but solvable.

I think you are dead on with this one, you have an advantage in that
no code has been written so mutithreading doesn't have to be worked
into the current code base.  I think in many cases it is harder to
convert single threaded applications into multithreaded apps than it
would be to just rewrite applications with multithreading.

>
>
> 2. IP Reputation Sharing
> I want to feed these gigs of data I have and other projects have into my
> security devices and let it use that data to make smarter decisions. IP
> reputation isn't a new concept, but applying it in realtime will be a
> challenge. But this also opens us up to the possibility of sharing
> reputation data between ourselves.
>
> Imagine clouds of peer organizations sharing ip reputation between their
> security devices. Each benefits from teh data gained and contributes
> back what they encounter. All organizations become more safe.
>
> Then imagine organizations that collect this data for a living. We have
> an avenue for this data to be more commercially viable.
>
I love this idea as well, but would not limit it to just IP data.  The
more data that is shared and collaborated on in a scoring system like
you mention further down in your post the better in my opinion.  I
would like to see a correlation engine (ie you scoring scheme) factor
in shared DNS data, Domain Reputations, ASN Reputations, ISP
Reputations, Port Statistics, Protocol Statistics, Threat
Indexes/Activities and statisitcs, vulnerability probabilities and
activities, Network/Application/OS awareness, and GEO type statistics.
 I know this is kind of a big dream to have widely dispersed
geographical networks sharing statistical data in real time that could
be correlated to ensure even the smallest networks obtained the
intelligence level and threat visibilities/awareness levels of the
largest networks in the cloud...  Obviously this is no trivial task!
What I do see is if this idea/dream became reality, you would see AI
implemented into IDS', Firewalls, Routers, Servers, Applications, and
anything else that could listen for these correlated data statistics
and adjust their configurations based off live data.  Imagine an
IDS/IPS that could auto magically load and unload specific rules to
meet the threats seen on other networks in anticipation of attacks to
come that may not have reached them yet or routers that could create
null routes for troubled/bad IP subnets based off of data intelligence
seen by this super cloud. Even web servers that could create mod
rewrite rules or acls to prevent exploit bots from delivering their
drive by badness, which I am sure have all grown to love in the last
six months...   I see limitless possibilities, but reality may play a
factor here ;)
>
> 3. Native ipv6
> Of course. No brainer there.
>
>
> 4. Native Hardware acceleration support
> There are a number of hardware acceleration technologies that could be
> more effectively built into the engine from the start, versus the
> back-asswards reverse engineering we have to do now to effectively
> accelerate.
>
Really cool idea and would like to see this go forward, but again not
a trivial mission.  With hardware acceleration you must factor in
cost.  Why do we spend hours hacking the kernel, libpcap, and
applications we use?  Easy cause most of us are under budget
constraints and can not afford to implement many of these hardware
accelerators...    I know we are going through the 10 Gb issues, and
well buying a Bivio and Network General sniffer to do this is very
painful for us and will break the bank.... that being said I welcome
competitors in this area as I believe competition could cause some of
these hardware acceleration technologies to fall in price.
>
> 5. Scoring
> Spam-assassin style point scoring. This would go a long way to
> eliminating false positives. The absolutely sure 100% guaranteed true
> positive rules of course would still hit. But the ones that are wrong as
> often as right could be given a score, say a half a point. If something
> else happens from that host within a certain timeframe that pushes that
> over a threshold then all of these alerts come back and can be acted
> upon with more confidence they're real. Complicated, but worthwhile.
>
>
>
> OK, those are my initial wish list items. Who has more? What else should
> we do? Any problems with the above?
>
> Matt
>
>
>
> --
> --------------------------------------------
> Matthew Jonkman
> Emerging Threats
> Phone 765-429-0398
> Fax 312-264-0205
> http://www.emergingthreats.net
> --------------------------------------------
>
> PGP: http://www.jonkmans.com/mattjonkman.asc
>
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>