[Discussion] Features and Design Strategy

Fri Oct 17 21:56:20 UTC 2008

Yes, yes, and yes.  This is the part where a modular, pluggable architecture
lets everyone contribute their various scoring mechanisms and checks.  There
could be a plugin for your SIP/DIP methodology, a plugin for the Google
malware domain checker, to Shadowserver.org, etc., etc.  But the key is that
we don't have to make a web request on a per-packet basis, rather, we update
lists for the sniffer periodically, and we only have to check external
sources when we are handed an event from the sniffer.  If we buffer the data
from the sniffer to the event policy engine queue, then we can afford to do
lots and lots of scoring, checking, re-checking, and ranking before a human
ever has to wonder what's going on without dropping packets.  As an added
bonus, the source lists can be updated based on the collective decisions, so
we get a positive cycle of pruning the source lists.

Thanks for the feedback!  I can't wait to all of the cool stuff this group
is going to dream up.

Martin

On Fri, Oct 17, 2008 at 4:13 PM, <robert.jamison at bt.com> wrote:

> Great e-mail Martin!
>
> Disclaimer: My comments and thoughts are coming from an MSSP
> perspective, not from a local network security administrator with strong
> knowledge of and view into a given environment (this is good and bad, I
> suppose...).
>
> Expanding on Martin's comments:
>
> " So, while perl would be far
> > too slow to do real-time content matching, it is certainly not too
> > slow to receive an alert, check to see if this source IP has alerted
> > recently, create an incident with some priority, see who needs to be
> > notified of the incident, and then send an alert."
>
> Checking to see if the SIP has alerted recently, and checking for
> derogatory information on the DIP gives opportunity to consider
> additional evidence outside the context of the observed session.  While
> a blacklist provides a seed to begin session tracking, it's typically
> Boolean and doesn't provide historical or concurrent significance in
> establishing an event priority.  The common framework is building on
> that by more effective session analysis and intelligent classification,
> resulting in better incident response--I'm not going to add to this as
> my perspective is much more lateral than focused.
>
> One of the projects I've been working on is the analysis and storage of
> firewall logs from functionally and geographically diverse environments.
> We are attempting to leverage cross customer firewall accept and deny
> data as well as security events from IDS/IPS systems which may or may
> not be on the same networks in which the firewalls exist.  This could
> perhaps be viewed as filling gaps in a protective perimeter by
> harnessing detection resources that exist elsewhere.  Parallelism vs.
> Serialism.    This is to build confidence that a customer client is
> indeed infected, before reaction policy is even engaged.  Even though
> SIP or client 'Alpha' may only be seen from a single network (which we,
> as a MSSP typically don't have control over) security alerts from
> disparate IDS/IPS platforms often implicate the same destination address
> as being a control node, or hosting malware.  Some of the input we aim
> to feed the incident priority algorithm are:
>
> 1) How many other networks have connections to this blacklisted DIP
> 2) What ports have been accessed
> 3) What is the historical frequency of access to this DIP from all
> monitored environments
> 4) What are the IDS/IPS signatures which have been tripped
> 5) Is there a common security classification of those sigs
> 6) Has any forensic investigation of a client that has been in contact
> with this DIP uncovered a rootkit or malware instance
> 7) What is the historic significance in IP addresses in the DIP's
> allocated block (e.g. have other addresses in that block been listed as
> DIPs in other security alerts within the same or other attack
> classification type?)
>
> I hope to participate more in this project.
>
> Thanks,
>
>
> Rob Jamison
>
>
>
> -----Original Message-----
> From: discussion-bounces at openinfosecfoundation.org
> [mailto:discussion-bounces at openinfosecfoundation.org] On Behalf Of Matt
> Jonkman
> Sent: Friday, October 17, 2008 3:58 PM
> To: Martin Holste
> Cc: discussion at openinfosecfoundation.org
> Subject: Re: [Discussion] Features and Design Strategy
>
> Well put Martin. I definitely like the idea. It goes well with one of
> the core things we need to do, let everything thread off away from the
> flow of packets as possible.
>
> So to generalize a bit more, we should strive to make anything possible
> able to be hooked, or fed out to a side process as possible. Output of
> course, but anything else as well. Just leave generic hook points any
> process could pull data from fifo style.
>
> This would likely be where we could let the extracted binaries come out
> of the core processes as well.
>
> Great concept to add, I'll get this into the wiki so we don't lose the
> idea.
>
> What challenges does anyone see in this concept?
>
> What other advantages might we gain if we plan correctly?
>
> Matt
>
> Martin Holste wrote:
> > First off, many congrats, Matt!
> >
> > When you look at the threat landscape, it's clear that the most
> imminent
> > threats are against web-facing clients and servers, so I think we
> should
> > focus on client-side attacks and attacks against web applications.
> > Detecting SMB protocol attacks, etc. is still important, but most
> > attacks against non-web protocols need a foothold on the LAN first.
> > Most IDS are historically built for defending servers from clients,
> and
> > I think that idea is becoming a bit antiquated, though certainly far
> > from obsolete.
> >
> > My group has had great success doing what John at Berkeley is/will be
> > doing with file carving.  We've been auto-extracting packed files from
> > the network stream and auto-submitting to VirusTotal for over a year
> > now.  John's point about processing the layer-7 data inside the
> streams
> > is dead-on.  We need to move away from answering "what strings did the
> > traffic contain" to answering "what did the host actually do?"  We
> also
> > need to keep in mind that canonically defining what "bad" behavior is
> > can be incredibly difficult even if you're just trying to put it in
> words.
> >
> > The design strategy that it is important to me is plugin
> extensibility.
> > One of the reasons I love perl so much is that there is a CPAN module
> > for just about anything you could ever dream of needing.  The extreme
> > ease with which you can incorporate others' code into your own project
> > is what has kept it thriving for so long.  The differences between
> > individual organization's security needs and preferences mandate that
> a
> > successful project will have a long-term goal of being extremely
> > modular, while also having a core functionality that is superb.
> >
> > Since Snort is what I'm familiar with, I will try to make my point by
> > comparison:  Snort has excellent core functionalilty, but despite
> > Marty's best efforts, extending it is not a trivial task because the
> > code is in C.  When Jason Brevenik posted his SnortUnified.pm module
> > which gives perl access to realtime Snort alerts, I realized that
> there
> > was suddenly an avenue for decoupling the pure network-grepping core
> > functionality of Snort with the action component.  Moreover, I now had
> a
> > hook for writing extremely customized and complicated event handling
> in
> > my programming language of choice instead of trying to debug segfaults
> > in preprocessors all day.  So, while perl would be far too slow to do
> > real-time content matching, it is certainly not too slow to receive an
> > alert, check to see if this source IP has alerted recently, create an
> > incident with some priority, see who needs to be notified of the
> > incident, and then send an alert.  So, the point is, we only need the
> > really fast component to do a limited number of tasks, and if we
> > decouple the real-time, per-packet tasks from the high level "what do
> we
> > do with this" tasks, anyone is free to trivially write their own hooks
> > into how they want things handled, which is the part that varies the
> > most from org to org.  Marty recognized all of this when he wrote the
> > unified output module, but Sourcefire never had a commercial reason
> for
> > extending it much since it would encroach on their own product
> > offerings.  OISF has the advantage of not being for-profit, and so we
> > can make extensibility a priority.
> >
> > What I would like to see is a brand-new program with functions similar
> > to Snort's HTTP preprocessor which has the sole purpose in life of
> > sniffing the network and attaching HTTP header fields to the traffic,
> > most importantly the host, content-type, and content-length fields.
> > This daemon would read a config file which specifies rules more like
> > scripts.  The rules would have variables populated from a policy
> server,
> > and it would update these variables very regularly (without a program
> > restart of any kind).  This would de-couple the work of sniffing from
> > the work of policy setting.  The policy server would get its variables
> > from all kinds of sources, like remote XML files.  Armed with this,
> very
> > simple signatures which identify web applications could be written.
> > Here's a pseudo-code example of what I think the configs would look
> like:
> >
> > ## On the policy server ##
> >
> > # Declare vars
> >
> > DECLARE $attacked = ARRAY;
> >
> > DECLARE $attackers = ARRAY;
> >
> > DECLARE $bad_hosts = { type=dns,
> url=blacklists.example.org/bad_dns.xml
> > <http://blacklists.example.org/bad_dns.xml>, refresh_list=30 };
> >
> > DECLARE $bad_ips = { type=ip,
> > url=more_blacklists.example.org/bad_ips.xml
> > <http://more_blacklists.example.org/bad_ips.xml>, refresh_list=30 };
> >
> > DECLARE $our_windows_hosts = { type=ip,
> > url=http://mycompany.example.org/our_windows_hosts.xml,
> refresh_list=900 };
> >
> > ## On the HTTP sniffer ##
> >
> > # Detect our web apps
> >
> > ID $google AS [http,https ] FROM [ host='*.google.com
> > <http://google.com>' ];
> >
> > ID $google_search_result AS [ $google ] MATCHING [ uri =~ /search/ ];
> >
> > ID $our_web_clients AS [http,https] TO [ $our_windows_hosts ];
> >
> > ID $us_sending_email AS [smtp] FROM [ $our_web_clients ];
> >
> > ID $malware_webapp AS [ http, https ] FROM [ $bad_hosts OR $bad_ips ];
> >
> > # Define our actions
> >
> > # Create an event and record a client as attacked if it is referred to
> a
> > malware page by a Google search
> >
> > FORWARD EVENT IF { $google_search_result REFERS $malware_webapp } AND
> [
> > ADD $_CLIENT TO $attacked, ADD $_SERVER TO $attackers ];
> >
> > The important thing is that our daemon doing the HTTP sniffing is
> > agnostic in that it has no idea if things are bad or good, it just
> lets
> > somebody know when a predefined set of criteria has occurred.  It is
> up
> > to an event policy server to receive these events and do something
> > intelligent (and highly configurable) with it.  So, we've decoupled
> any
> > decision making from the component that has the most time-sensitive
> work
> > to do.  All of the other components can buffer their work, but the
> > network sniffer needs as much work lifted from it as possible so that
> it
> > can cope with high loads.  The policy server components would be the
> > only parts that would require much customization, and they would be
> > written in a high-level language like perl for maximum extensibility.
> >
> > Let's also not forget the plethora of other apps out there that we can
> > leverage.  Of particular importance would be SANCP and PADS.
> >
> > SANCP's newest version contains a feature that John Curry added for me
> > which writes out an index of the file position within a bulk pcap
> where
> > a packet starts and stops.  This means that you can do instantaenous
> > bulk packet retrievals using the index.  In my tests, I can pull an
> > arbitrary host's traffic from a 30 GB pcap in under 5 seconds.  This
> > technique could be used for file carving, which would provide the
> > ability to send to sandboxes, send to VirusTotal, Symantec, etc.
> >
> > PADS provides a ready-made daemon for identifying the protocols used
> > between hosts.  If PADS fed a policy server, we could automagically
> > update the policy server when new Windows hosts come online, or when
> an
> > SMTP session starts.  Since the HTTP sniffer would be reading from the
> > policy server on a regular basis, PADS would be able to keep it
> informed
> > as to the state of the network.  P0f could work as well.
> >
> > So, the main idea would be to have a very lightweight daemon with a
> > specific task forward events to an event policy server written in a
> > higher level language, and to remove almost all decision making from
> the
> > sniffer.
> >
> > Comments?
> >
> > Thanks,
> >
> > Martin
> >
> >
> >
> ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Discussion mailing list
> > Discussion at openinfosecfoundation.org
> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
> --
> --------------------------------------------
> Matthew Jonkman
> Emerging Threats
> Phone 765-429-0398
> Fax 312-264-0205
> http://www.emergingthreats.net
> --------------------------------------------
>
> PGP: http://www.jonkmans.com/mattjonkman.asc
>
>
> _______________________________________________
> Discussion mailing list
> Discussion at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081017/363e5ef7/attachment-0002.html>