[Discussion] features (mainly dns)

Sat Dec 6 12:43:09 UTC 2008

On Tue, Nov 18, 2008 at 10:35:34AM -0500, Matt Jonkman wrote:

> >>       * DNS responses which had a low to very low TTL (time to live)
> >>         value, which is somewhat unusual;
> 
> We tried sigs for this a while ago and found that there are as many
> legit low ttl responses as there were hostile. The sigs were reliable
> and relatively low load, but the information wasn't actionable
> unfortunately. Pretty everyone with akami hit on it, etc.

Years ago this was a useful filtering heuristic.  E.g., see this 2005
plot, showing the percentage of ~500 sampled bot C&Cs with various low
TTLs:

   http://fritz.cc.gt.atl.ga.us/ttl_cdf.png

But these days, low TTLs are quite commonly used for web 2.0 and large
distributed CDNs.  ('Cloud computing' need network agility it seems.)
Can you whitelist these low-TTL sites faster than they appear?  The
problem is that top sites change their CDN domains often.

I've also noted that bot C&Cs artificially increased their TTLs around
the time news of this filter became more popular (starting 2005 it
became public, if I recall).  You'll still find low TTLs, but some
even use 86400.

Low TTLs are also used in server migration.  It would be interesting
if your tool kept state, and could be config'd to actually observe such
migrations.  (You'd find flux and white listable migrations; but then
repeated migrations would remove the white listables and reveal the
flux.)

> >>       * DNS responses which contained a domain that belonged to one of
> >>         a long list of dynamic DNS providers;
> 
> That's an interesting one. A good way for us to use DNS reputation adata
> that we'd surely collect alongside IP reputation data.

In my opinion, this was once a good heuristic.  But over the years,
many of the DDNS providers have *aggressively* cleaned up their zones.
After years of ddos wars, many botmasters got the message and moved on
to other dynamic providers (often outside the US), or to places with
poor abuse handling.  (Or they fluxed, started their own criminal DNS
farms, etc.)  Many dyndns providers also provide services for large
companies these days; the industry has grown up out of the garage.
(Look at David U's old ddns; only the foolish/naive botmasters try to
abuse his ddns service.)

So, if you use this factor, you might rank those zones by the DDNS's
reputation, responsiveness, etc.  Just flagging the DDNS origins of a
zone alone is too simple, and does not recognize the remarkable clean
up effort this industry took on, often alone.

> >>       * DNS queries which were issued more frequently by the client
> >>         than would be expected given the TTL for that hostname;
> 
> How do you mean? Loke looking for a client that's making repeated dns
> queries within the TTL? Maybe poorly coded bots?

I'd say you have a winner here; this is a single class classifier,
almost.  Your false positives are now just NAT, much of the Pacific
rim, etc.  But this is a very good criteria; the state costs for
tracking can be expensive, however.  (See Barford's paper, below for
an idea).

> >>       * DNS requests for a hostname outside of the local namespace
> >>         which were responded to with a resource record pointing to an
> >>         IP address within either 127.0.0.0/8, 0.0.0.0/32, RFC1918 IP
> >>         space, or anywhere inside the public or private IP space of
> >>         the organization;
> 
> Absolutely here. But would require a good deal of local config, but
> worthwhile I think.

RRsets that match DNSBL listed hosts make an *excellent* intel feed.
(Same with LHS BLs, but I've seen other people who already want this
fed in your tool.)

RRsets pointing to PBL country would also be a good heuristic for
corporate networks; your false positives here are soft FPs, since they
pose a possible policy issue if not an infection.  For example, these
are often workers looking at legitimate vanity DDNS sites on the
company dime.  See the note above, however, about the growing
professional use of ddns services.

Your best bet, I think, is to review RFC 1912 and write filter rules
for "violations" of each suggestion (it's not standards track; these
are factors not filters).  Some of this would require an active
probing backend--something best shared or obtained through a feed
using passive data collection (SIE, etc.).

But these are routinely violated by bots not using ddns or
professional DNS hosting.   Look for:

  -- missing glue in parent zones (esp. if the parent is legit, a TLD)

  -- missing stealth; ns listed at the authority, but missing in the
     parent.  Such deployments are risky propositions (relying on rfc
     2181 inclusion of authorities in answers, which may or may not
     happen).

  -- NS in the parent not listed as NS in the authority.  (Harmless;
     sometimes configuration, but bot masters are often not complete.)

  -- SOA Refresh/Retry/Expire/Min values are low; this is flux-ish.
     But see my notes above about how the botmasters became educated
     about low TTLs.  Low SOA Refresh values in particular are very
     odd, since masters see many more zone xfers; that's an unusual
     arrangement; it suggests double flux.  Flag instances of Retry <<
     Refresh, or where these ratios are not in balance.

     Extremely large refresh values are odd (sometimes serial number
     fields using epochs, mistyped into the refresh field); perhaps
     yellow flag those above 3D.  (Some are MAXINT.)  This is not
     useful for attackers, but your tool may note this for potential
     staleness in the secondary.

     Excuse the ascii art, but here's a distribution of 1 million such
     authority refreshes, mostly white listables:

       http://fritz.cc.gt.atl.ga.us/soa_refresh_dist.txt

     The domains in the tails of this dist. are always interesting;
     I'd like your tool to alert on deviations of these and other
     fields.

  -- Invalid RNAMEs in SOA are unfortunately common, and (for obvious
     reasons) often not in the zone.  Some TLDs require the MNAME to
     contain the hostname in the first NS; I'm not sure how often
     that's violated, but you could find out by giving a config for
     warning/alerts.  And people forget to escape '\.' for compound
     email addresses.  But I would flag those that are obviously
     non-starters (e.g., *.localhost, invalid TLDs, etc.).

     RFC 2142 suggests reserving hostmaster@ for this; but otherwise
     this area is in complete chaos.  Here's how the top 1 million
     domains on the Internet handled this (which gives an idea of the
     amount of misconfiguration out there):

      http://fritz.cc.gt.atl.ga.us/rnames_for_top1M_sites.txt

     (Note that some of these are for malicious domains; this is not a
     white list.)  This list is not complete noise; look at the bottom
     of the list for amusing examples; some in there are malicious.

     A config that just lets me flag 2142 violations would be too
     noisey; perhaps some more heuristics would let one alert on bogus
     contact info (which, while common, may reduce confidence in
     domains).

  -- missing L7/web2.0-ish records; e.g., no www.*, no MX, etc.

  -- while common in legit hosting, I'd appreciate your future sensor
     spitting out an alert on NS that fail to allow TCP fallback.  All
     these marginal, low end DNS widgets that barely speak DNS are a
     barrier to 0x20 adoption, DNSSEC, SPR, etc.  They're cheap, but a
     form of DNS blight, IMHO; they're insecure and unpatchable, and
     break secure appliances behind them.  I recommend listing them
     with a yellow caution flag to encourage their replacement with
     full DNS/DNSSEC speakers.  (This is a political suggestion
     obviously; but please consider this idea.)

  -- Non-matching serials in primary and secondaries.  Actually, this
     is done by Microsoft intentionally for signaling in active
     directory.  But most DNS servers would be BIND, where this is a
     misconfig.  Yellow flag.  Or use this as a fingerprint for a
     windows host, and adjust your scrutiny of the associated traffic.

Also, if your sensor can please track recursives differently than
authorities, it would be useful to check the port randomization
behavior of the recursives.  I.e., if std dev == 0, it's a flag that
in this post-Kaminsky era, cached answers from those recursives are
trivially poisoned, and may need more aggressive checking.  No single
packet will reveal this (well, a single resolution could be forced
through many cuts to yield SPR info).  The point is you'll need to
keep state in the tool.  This is DNS reputation, a variation on IP
reputation.

Finally, I'll suggest one more heuristic that finds poison quite well:
RTT deltas.  We're finishing a study of inline poisoning detection;
RTT deltas are an excellent filter, and they're very difficult to
evade.  (And this even catches corruption of the zone or authority
breakins, like the recent checkfree incident.)  The downside is that
state costs for per-domain RTT tracking can be large.

So how will you store all this data I want?  I'd suggest looking into
novel DNS-specific data structures.  Paul Barford has a good one,
which is useful for all of the ideas noted above and more:

    http://www.cs.wisc.edu/~pb/imc08.pdf

That's quite a list above, and I could add much more.  Perhaps you'd
have a module/API that would let users write their own heuristics to:

  -- import their own feeds of DNS reputation;

  -- pull common BLs (DNSBLs and Dom BLs), and store them efficiently;

  -- some statistical checks, e.g., so I could identify covert
     channels in DNS, and such.

I'm looking forward to this project.

-- 
David Dagon              /"\                          "When cryptography
dagon at cc.gatech.edu      \ /  ASCII RIBBON CAMPAIGN    is outlawed, bayl
Ph.D. Student             X     AGAINST HTML MAIL      bhgynjf jvyy unir
Georgia Inst. of Tech.   / \                           cevinpl."