[Discussion] Features

Fri Oct 17 08:38:58 UTC 2008

Hi everyone,

Andre Ludwig wrote:
> Since this is an "ideas" thread and not "comment on ideas" thread ill 
> keep it short and blunt.
>
> I left a few things out of the "why ids sucks" thread on purpose, one of 
> which i will address here.
>
> 7. Ability to look across packets in a stream, be it 3 packets before or 
> 4 packets ahead.  The ability to follow conditions of state that are 
> triggered by packets before or after a "match" would be extremely 
> valuable. (rpc comes to mind, as well as some application layer 
> attacks)   Ideally the ability to perform functions against packets (be 
> it arithmetic,  comparisons, or functions on "running variables" for a 
> "attack detection set").  The concept here of course is to follow a tcp 
> stream looking for precursory actions that are needed to finesse an 
> endpoint into an exploitable situation.  Be it forcing the endpoint to 
> allocate a buffer of X size with XYZ rpc call, that is later exploited 
> by calls to 3 other RPC calls with varying parameters.  The critical key 
> is to engineer a system that affords you the flexibility to effectively 
> "emulate" and follow increasingly complex attack scenarios to the point 
> of detecting actual attacks and not simply triggering a FP.   My single 
> largest frustration with snort (beyond the fact that i absolutely hate 
> its limited ability to detect "proper attacks") is that you are 
> effectively looking at one packet at a time.  You do not have the 
> ability to create per tcp stream variables that would allow you to 
> compare against packets furthur down the stream. 
>
>   
One thing we've been thinking about is something like Snort's flowbits,
but then way extended. Having a way to capture data from the
packets/stream, setting and calculating with counters, etc all per
packet, flow/stream, host, network and globally. And then provide the
rule language to compare each and everyone of those with each other, do
translations etc. ModSecurity has features like this and they are very
useful.

For example in ModSecurity you can match on the password on HTTP basic
auth by capturing the string containing the username and password from
the HTTP header, splitting it, decoding it's base64. All from the rules
language.

Technically very challenging I'm sure, but lets keep the discussion on
what we want in an ideal world and worry about how to do implementations
that perform well in the real world later :P

Regards,
Victor

> 8.  Some sort of meta language needs to be created that easily and 
> effectively can communicate any applications "state".  Even if this 
> means creating application specific "translation" modules that allow end 
> users to effectively outline functions used by an application (and what 
> they do).  This of course would come in handy by allowing end users the 
> ability to effectively "peer" inside the "mapped logic" of an 
> application.  This in turn (if done right) could aid in an analysts 
> ability to alert and investigate malicious behavior of an application vs 
> simply producing mind numbingly general "signatures' and hoping for the 
> best.  (xss alerts come to mind, as well as other web app attacks)
>
>
> Jeremy wrote:
>   
>> Comments are inline but first I would like to add an item:
>>
>> 6.  Network intelligence through passive identification.  What I mean
>> by this is like Snorts RNA, NMAP, and p0s fingerprinting to
>> dynamically update inventory data for correlation engine usage.  I
>> think we all know how great it would be to have this data which could
>> dynamically or auto magically configure/reconfigure signatures and
>> correlation engines to align with applications and operating systems
>> to truly substantiate the risk of attacks and/or threats.  Having the
>> availability of correlated network intelligence could make tremendous
>> headway in the world of anomaly detection i.e.  a windows xp home
>> computer correlated with network flow data and port data could easily
>> be alerted on for any outgoing port 25 connections, as they are most
>> likely the newest member of a spam bot network.  Obviously this is
>> just a trivial example but it still is not a trivial task to pull off
>> even with the most advanced SIM.  This would also allow for
>> operational performance enhancements by providing context regarding
>> the network they operate in.  The dynamic configuration and
>> reconfiguration of network devices to ignore threats that are not
>> applicable would definitely enhance performance by reducing overhead
>> caused by signatures and other security measures.
>>
>> On Thu, Oct 16, 2008 at 8:00 PM, Matt Jonkman <jonkman at jonkmans.com> wrote:
>>   
>>     
>>> Here's the big thread. And don't be afraid to start sub-threads for
>>> specifics here.
>>>
>>> The features we want to go after here are the primary reason we sought
>>> this funding and are taking this challenge on. Existing stuff works, but
>>> there's SO much more we could be doing by looking past traditional ips
>>> strengths. The challenge is that those things aren't conducive to making
>>> a commercial product with millions invested in development. No one can
>>> take this risk now, so we're going this route to make it happen.
>>>
>>> We have information about bad guys, bad places, and bad patterns. Lots
>>> of it, terabytes of it. We've got gigs of data about bad stuff in the
>>> sandnet at emerging threats alone. But most of that we can't effectively
>>> act upon. We can't give huge lists of bad IPs to most tools, we can't
>>> feed behavior patterns to existing tools, we can't share scan data
>>> globally, etc.
>>>
>>> So here we are. I have things I wish I could do, you have things you
>>> wish you could do, over the next couple of months we aim to get to the
>>> core set of the most important things that most of us want to be able to
>>> do. Then we'll go after it.
>>>
>>> So here's my wish list:
>>>
>>> 1. Native multithreading.
>>> Not each preprocessor or post processor can go to a thread, but each
>>> stream can take a thread. Think apache. More servers = more requests
>>> served. THe complications of sharing state between them and the like is
>>> a challenge, but solvable.
>>>     
>>>       
>> I think you are dead on with this one, you have an advantage in that
>> no code has been written so mutithreading doesn't have to be worked
>> into the current code base.  I think in many cases it is harder to
>> convert single threaded applications into multithreaded apps than it
>> would be to just rewrite applications with multithreading.
>>
>>   
>>     
>>> 2. IP Reputation Sharing
>>> I want to feed these gigs of data I have and other projects have into my
>>> security devices and let it use that data to make smarter decisions. IP
>>> reputation isn't a new concept, but applying it in realtime will be a
>>> challenge. But this also opens us up to the possibility of sharing
>>> reputation data between ourselves.
>>>
>>> Imagine clouds of peer organizations sharing ip reputation between their
>>> security devices. Each benefits from teh data gained and contributes
>>> back what they encounter. All organizations become more safe.
>>>
>>> Then imagine organizations that collect this data for a living. We have
>>> an avenue for this data to be more commercially viable.
>>>
>>>     
>>>       
>> I love this idea as well, but would not limit it to just IP data.  The
>> more data that is shared and collaborated on in a scoring system like
>> you mention further down in your post the better in my opinion.  I
>> would like to see a correlation engine (ie you scoring scheme) factor
>> in shared DNS data, Domain Reputations, ASN Reputations, ISP
>> Reputations, Port Statistics, Protocol Statistics, Threat
>> Indexes/Activities and statisitcs, vulnerability probabilities and
>> activities, Network/Application/OS awareness, and GEO type statistics.
>>  I know this is kind of a big dream to have widely dispersed
>> geographical networks sharing statistical data in real time that could
>> be correlated to ensure even the smallest networks obtained the
>> intelligence level and threat visibilities/awareness levels of the
>> largest networks in the cloud...  Obviously this is no trivial task!
>> What I do see is if this idea/dream became reality, you would see AI
>> implemented into IDS', Firewalls, Routers, Servers, Applications, and
>> anything else that could listen for these correlated data statistics
>> and adjust their configurations based off live data.  Imagine an
>> IDS/IPS that could auto magically load and unload specific rules to
>> meet the threats seen on other networks in anticipation of attacks to
>> come that may not have reached them yet or routers that could create
>> null routes for troubled/bad IP subnets based off of data intelligence
>> seen by this super cloud. Even web servers that could create mod
>> rewrite rules or acls to prevent exploit bots from delivering their
>> drive by badness, which I am sure have all grown to love in the last
>> six months...   I see limitless possibilities, but reality may play a
>> factor here ;)
>>   
>>     
>>> 3. Native ipv6
>>> Of course. No brainer there.
>>>
>>>
>>> 4. Native Hardware acceleration support
>>> There are a number of hardware acceleration technologies that could be
>>> more effectively built into the engine from the start, versus the
>>> back-asswards reverse engineering we have to do now to effectively
>>> accelerate.
>>>
>>>     
>>>       
>> Really cool idea and would like to see this go forward, but again not
>> a trivial mission.  With hardware acceleration you must factor in
>> cost.  Why do we spend hours hacking the kernel, libpcap, and
>> applications we use?  Easy cause most of us are under budget
>> constraints and can not afford to implement many of these hardware
>> accelerators...    I know we are going through the 10 Gb issues, and
>> well buying a Bivio and Network General sniffer to do this is very
>> painful for us and will break the bank.... that being said I welcome
>> competitors in this area as I believe competition could cause some of
>> these hardware acceleration technologies to fall in price.
>>   
>>     
>>> 5. Scoring
>>> Spam-assassin style point scoring. This would go a long way to
>>> eliminating false positives. The absolutely sure 100% guaranteed true
>>> positive rules of course would still hit. But the ones that are wrong as
>>> often as right could be given a score, say a half a point. If something
>>> else happens from that host within a certain timeframe that pushes that
>>> over a threshold then all of these alerts come back and can be acted
>>> upon with more confidence they're real. Complicated, but worthwhile.
>>>
>>>
>>>
>>> OK, those are my initial wish list items. Who has more? What else should
>>> we do? Any problems with the above?
>>>
>>> Matt
>>>
>>>
>>>
>>> --
>>> --------------------------------------------
>>> Matthew Jonkman
>>> Emerging Threats
>>> Phone 765-429-0398
>>> Fax 312-264-0205
>>> http://www.emergingthreats.net
>>> --------------------------------------------
>>>
>>> PGP: http://www.jonkmans.com/mattjonkman.asc
>>>
>>>
>>> _______________________________________________
>>> Discussion mailing list
>>> Discussion at openinfosecfoundation.org
>>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>>>
>>>     
>>>       
>> _______________________________________________
>> Discussion mailing list
>> Discussion at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>>
>>   
>>     
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>   