From jonkman at jonkmans.com Wed Dec 3 17:40:58 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Wed, 03 Dec 2008 17:40:58 -0500 Subject: [Discussion] Features - Designing for Scaleability In-Reply-To: References: Message-ID: <49370AFA.5020404@jonkmans.com> So you're thinking the tool should have it's own management protocol to a central hub built in. To allow sensors to be relatively autonomous, pulling config and reporting data through a single protocol? Matt Martin Holste wrote: > John, > > I think you've hit on some key points here. I manage about a dozen > sensors, and there is a definite line that is crossed somewhere around a > half-dozen servers where managing the configuration and health of the > boxes requires some power tools. I prefer Nagios for general system > health monitoring, so that has never been an issue. Conversely, it has > been an incredible challenge to manage rules and granular Snort > performance on many boxes with multiple Snort instances. I've taken the > route you have with many home-grown scripts, etc., but there's always > something new that comes out with Snort making the management > difficult. Specifically, dealing with SO rules throws a huge wrench in > the works when dealing with mixed architectures. > > My strategy thus far has been to have my sensors completely self > contained and manage them centrally using a home-grown system written in > Perl to make generic queries and issue commands. The databases which > Snort writes to are on the box (performance really isn't impacted at all > by doing this). The Perl agent receives queries like "get all alerts in > last X minutes," or "add this rule to the running configuration," or > "calculate stats for Mbps and alerts per second for last 1000 ticks." > The key is that since all the data is on the sensor, it scales very > well. The central Perl client can then parallelize queries so all > sensors can search at the same time much faster than if all the alerts > were located in one central database. Since everything goes through the > central management client, it can easily log all the actions (and even > queries) which are run for audit/change management purposes. > > For encryption, I run everything through OpenVPN. This works really > well, especially for debugging, since it is much easier to tcpdump a > tunnel interface than get a hook into a TLS socket. > > I'm working on a 2.0 version of this architecture which will be entirely > asynchronous, so that the sensors can alert the central management hub > on predetermined criteria. > > For truly large deployments, I think you're right that a parent-child > setup might be necessary. An exploration of peer-to-peer techniques > might be an interesting, but I think that for simplicity, a tree > hierarchy would make the most sense. That is, a "hub" may have X number > of servers assigned to it, and a hub higher up the tree would be able to > ask hubs to delegate queries down the hierarchy. It would be > interesting to see if there would be performance gains versus attempting > parallelized queries over thousands of sensors from just one hub. My > thinking is that some of the sensors could also function as hubs. > > I think that we need to remember that the vast majority of users do not > deploy more than a few sensors, so we need to guard against spending too > much devel time on features that will only serve a small percentage of > the community. That said, audit, change management, archiving, and > other management features are things that benefit everyone. As long as > we keep it all modular, users can mix and match to get the features they > require. > > --Martin > > On Tue, Nov 18, 2008 at 12:47 PM, John Pritchard > > wrote: > > Team, > > My apologies if I've missed this being covered previously. Or, if the > Bro framework already takes some of these things into consideration. > > I'd like to suggest that we take very large deployments into > consideration when designing our solution. The kind of problems you > encounter when managing an infrastructure with a handful or even a > dozen different IDS sensors is very different than trying to > efficiently and consistently manage infrastructures with larger > footprints (e.g. > 100+ sensors). > > A couple of suggestions to help our design address these potential > deployment and management scenarios: > > 1) Centralized sensor and policy management platform (or framework) > --> Such a solution may be restricted to a single centralized server > or multiple servers. > --> Might be a parent -> child relationship among configuration > servers to segregate business units, or simply replication among > servers for disaster recovery / business continuity purposes > --> efficient, repeatable, and audit-able methodology for making > changes to both individual sensors as well as pre-defined groups of > sensors (e.g. dmz sensors, dns sensors, development lab sensors, > etc...) > --> My experience to date has been performing these kind of tasks with > home-grown scripts, ssh, scp, audit logs, etc... However, it would be > nice to find something a little more mature for this project. > > I have not used it, but there is a project called "Puppet" that looks > like it might be a good candidate for trying to develop a framework > along these lines: > http://reductivelabs.com/trac/puppet/wiki/PuppetIntroduction > > > 2) Centralized sensor and policy monitoring platform (or framework) > --> This is similar to the "management framework" concept, but the > focus is more on device health monitoring... and possibly policy > integrity monitoring... > --> For this piece, I'm thinking of something that provides functions > such as looking at memory utilization, cpu utilization, hard-drive > space, network interface stats... and other "bits" such as dates and > checksums for critical configuration files changed (e.g. detection > policies, tuning policies, variable definitions, etc)... > > There are a number of open-source enterprise monitoring utilities out > there. Here's one I've been playing with recently: > http://www.zabbix.com/ > > > 3) Distributed Data Repositories > I know we briefly touched on the database stuff when talking about > schema design. I just wanted to add a plug for a couple of things > here: > --> encrypted communication channels (sensor -> DB or sensor -> sensor) > --> ability to simultaneously log to 2 or more data repositories > > I strongly agree with the concept of designing modular solutions. So, > these kind of features can be "bolted on" if they were required. > > Look forward to everyone's thoughts on how we can most effectively > tackle problems of scale for large deployments. > > Cheers, John > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > ------------------------------------------------------------------------ > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Dec 5 13:02:49 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 05 Dec 2008 13:02:49 -0500 Subject: [Discussion] Update Message-ID: <49396CC9.4070506@jonkmans.com> An update on where we are and what's going on. Most of what's happening is behind the scenes. We've gotten our contracts and financials setup with DHS. Our accounting infrastructure is setup and functioning (that was a load of fun) as well. We are in discussions with the Software Freedom Law Center (the folks that brought you the GPL among many other things) to setup the consortium bylaws and framework so we can begin to cement the formal relationships with our sponsors and contributors. I believe we have a very ambitious feature list on the wiki. Thanks to all for your contributions and discussion so far. Please keep it going, especially I think in the areas of statistical analysis. That's something we have to concentrate on, and something I believe we're going to have to have outside (i.e. non-security geek) expertise to really do right. Please keep your eyes open for relevant research and talented statistically minded individuals we could bring into the project. Please keep the discussion going, and I'll update again soon once we have a draft consortium/board makeup ready for all to look at. Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From dagon at cc.gatech.edu Sat Dec 6 07:43:09 2008 From: dagon at cc.gatech.edu (David Dagon) Date: Sat, 6 Dec 2008 07:43:09 -0500 Subject: [Discussion] features (mainly dns) In-Reply-To: <4922E0C6.1090204@jonkmans.com> References: <1226959498.27961.3.camel@arodgers-panasonic> <4922E0C6.1090204@jonkmans.com> Message-ID: <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> On Tue, Nov 18, 2008 at 10:35:34AM -0500, Matt Jonkman wrote: > >> * DNS responses which had a low to very low TTL (time to live) > >> value, which is somewhat unusual; > > We tried sigs for this a while ago and found that there are as many > legit low ttl responses as there were hostile. The sigs were reliable > and relatively low load, but the information wasn't actionable > unfortunately. Pretty everyone with akami hit on it, etc. Years ago this was a useful filtering heuristic. E.g., see this 2005 plot, showing the percentage of ~500 sampled bot C&Cs with various low TTLs: http://fritz.cc.gt.atl.ga.us/ttl_cdf.png But these days, low TTLs are quite commonly used for web 2.0 and large distributed CDNs. ('Cloud computing' need network agility it seems.) Can you whitelist these low-TTL sites faster than they appear? The problem is that top sites change their CDN domains often. I've also noted that bot C&Cs artificially increased their TTLs around the time news of this filter became more popular (starting 2005 it became public, if I recall). You'll still find low TTLs, but some even use 86400. Low TTLs are also used in server migration. It would be interesting if your tool kept state, and could be config'd to actually observe such migrations. (You'd find flux and white listable migrations; but then repeated migrations would remove the white listables and reveal the flux.) > >> * DNS responses which contained a domain that belonged to one of > >> a long list of dynamic DNS providers; > > That's an interesting one. A good way for us to use DNS reputation adata > that we'd surely collect alongside IP reputation data. In my opinion, this was once a good heuristic. But over the years, many of the DDNS providers have *aggressively* cleaned up their zones. After years of ddos wars, many botmasters got the message and moved on to other dynamic providers (often outside the US), or to places with poor abuse handling. (Or they fluxed, started their own criminal DNS farms, etc.) Many dyndns providers also provide services for large companies these days; the industry has grown up out of the garage. (Look at David U's old ddns; only the foolish/naive botmasters try to abuse his ddns service.) So, if you use this factor, you might rank those zones by the DDNS's reputation, responsiveness, etc. Just flagging the DDNS origins of a zone alone is too simple, and does not recognize the remarkable clean up effort this industry took on, often alone. > >> * DNS queries which were issued more frequently by the client > >> than would be expected given the TTL for that hostname; > > How do you mean? Loke looking for a client that's making repeated dns > queries within the TTL? Maybe poorly coded bots? I'd say you have a winner here; this is a single class classifier, almost. Your false positives are now just NAT, much of the Pacific rim, etc. But this is a very good criteria; the state costs for tracking can be expensive, however. (See Barford's paper, below for an idea). > >> * DNS requests for a hostname outside of the local namespace > >> which were responded to with a resource record pointing to an > >> IP address within either 127.0.0.0/8, 0.0.0.0/32, RFC1918 IP > >> space, or anywhere inside the public or private IP space of > >> the organization; > > Absolutely here. But would require a good deal of local config, but > worthwhile I think. RRsets that match DNSBL listed hosts make an *excellent* intel feed. (Same with LHS BLs, but I've seen other people who already want this fed in your tool.) RRsets pointing to PBL country would also be a good heuristic for corporate networks; your false positives here are soft FPs, since they pose a possible policy issue if not an infection. For example, these are often workers looking at legitimate vanity DDNS sites on the company dime. See the note above, however, about the growing professional use of ddns services. Your best bet, I think, is to review RFC 1912 and write filter rules for "violations" of each suggestion (it's not standards track; these are factors not filters). Some of this would require an active probing backend--something best shared or obtained through a feed using passive data collection (SIE, etc.). But these are routinely violated by bots not using ddns or professional DNS hosting. Look for: -- missing glue in parent zones (esp. if the parent is legit, a TLD) -- missing stealth; ns listed at the authority, but missing in the parent. Such deployments are risky propositions (relying on rfc 2181 inclusion of authorities in answers, which may or may not happen). -- NS in the parent not listed as NS in the authority. (Harmless; sometimes configuration, but bot masters are often not complete.) -- SOA Refresh/Retry/Expire/Min values are low; this is flux-ish. But see my notes above about how the botmasters became educated about low TTLs. Low SOA Refresh values in particular are very odd, since masters see many more zone xfers; that's an unusual arrangement; it suggests double flux. Flag instances of Retry << Refresh, or where these ratios are not in balance. Extremely large refresh values are odd (sometimes serial number fields using epochs, mistyped into the refresh field); perhaps yellow flag those above 3D. (Some are MAXINT.) This is not useful for attackers, but your tool may note this for potential staleness in the secondary. Excuse the ascii art, but here's a distribution of 1 million such authority refreshes, mostly white listables: http://fritz.cc.gt.atl.ga.us/soa_refresh_dist.txt The domains in the tails of this dist. are always interesting; I'd like your tool to alert on deviations of these and other fields. -- Invalid RNAMEs in SOA are unfortunately common, and (for obvious reasons) often not in the zone. Some TLDs require the MNAME to contain the hostname in the first NS; I'm not sure how often that's violated, but you could find out by giving a config for warning/alerts. And people forget to escape '\.' for compound email addresses. But I would flag those that are obviously non-starters (e.g., *.localhost, invalid TLDs, etc.). RFC 2142 suggests reserving hostmaster@ for this; but otherwise this area is in complete chaos. Here's how the top 1 million domains on the Internet handled this (which gives an idea of the amount of misconfiguration out there): http://fritz.cc.gt.atl.ga.us/rnames_for_top1M_sites.txt (Note that some of these are for malicious domains; this is not a white list.) This list is not complete noise; look at the bottom of the list for amusing examples; some in there are malicious. A config that just lets me flag 2142 violations would be too noisey; perhaps some more heuristics would let one alert on bogus contact info (which, while common, may reduce confidence in domains). -- missing L7/web2.0-ish records; e.g., no www.*, no MX, etc. -- while common in legit hosting, I'd appreciate your future sensor spitting out an alert on NS that fail to allow TCP fallback. All these marginal, low end DNS widgets that barely speak DNS are a barrier to 0x20 adoption, DNSSEC, SPR, etc. They're cheap, but a form of DNS blight, IMHO; they're insecure and unpatchable, and break secure appliances behind them. I recommend listing them with a yellow caution flag to encourage their replacement with full DNS/DNSSEC speakers. (This is a political suggestion obviously; but please consider this idea.) -- Non-matching serials in primary and secondaries. Actually, this is done by Microsoft intentionally for signaling in active directory. But most DNS servers would be BIND, where this is a misconfig. Yellow flag. Or use this as a fingerprint for a windows host, and adjust your scrutiny of the associated traffic. Also, if your sensor can please track recursives differently than authorities, it would be useful to check the port randomization behavior of the recursives. I.e., if std dev == 0, it's a flag that in this post-Kaminsky era, cached answers from those recursives are trivially poisoned, and may need more aggressive checking. No single packet will reveal this (well, a single resolution could be forced through many cuts to yield SPR info). The point is you'll need to keep state in the tool. This is DNS reputation, a variation on IP reputation. Finally, I'll suggest one more heuristic that finds poison quite well: RTT deltas. We're finishing a study of inline poisoning detection; RTT deltas are an excellent filter, and they're very difficult to evade. (And this even catches corruption of the zone or authority breakins, like the recent checkfree incident.) The downside is that state costs for per-domain RTT tracking can be large. So how will you store all this data I want? I'd suggest looking into novel DNS-specific data structures. Paul Barford has a good one, which is useful for all of the ideas noted above and more: http://www.cs.wisc.edu/~pb/imc08.pdf That's quite a list above, and I could add much more. Perhaps you'd have a module/API that would let users write their own heuristics to: -- import their own feeds of DNS reputation; -- pull common BLs (DNSBLs and Dom BLs), and store them efficiently; -- some statistical checks, e.g., so I could identify covert channels in DNS, and such. I'm looking forward to this project. -- David Dagon /"\ "When cryptography dagon at cc.gatech.edu \ / ASCII RIBBON CAMPAIGN is outlawed, bayl Ph.D. Student X AGAINST HTML MAIL bhgynjf jvyy unir Georgia Inst. of Tech. / \ cevinpl." From fw at deneb.enyo.de Sun Dec 7 02:59:14 2008 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 07 Dec 2008 08:59:14 +0100 Subject: [Discussion] features (mainly dns) In-Reply-To: <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> (David Dagon's message of "Sat, 6 Dec 2008 07:43:09 -0500") References: <1226959498.27961.3.camel@arodgers-panasonic> <4922E0C6.1090204@jonkmans.com> <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> Message-ID: <87y6ys4drh.fsf@mid.deneb.enyo.de> * David Dagon: >> How do you mean? Loke looking for a client that's making repeated dns >> queries within the TTL? Maybe poorly coded bots? > > I'd say you have a winner here; this is a single class classifier, > almost. Your false positives are now just NAT, much of the Pacific > rim, etc. And all UNIX-like clients which do not perform local caching by default. 8-( From dagon at cc.gatech.edu Sun Dec 7 14:30:28 2008 From: dagon at cc.gatech.edu (David Dagon) Date: Sun, 7 Dec 2008 14:30:28 -0500 Subject: [Discussion] features (mainly dns) In-Reply-To: <87y6ys4drh.fsf@mid.deneb.enyo.de> References: <1226959498.27961.3.camel@arodgers-panasonic> <4922E0C6.1090204@jonkmans.com> <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> <87y6ys4drh.fsf@mid.deneb.enyo.de> Message-ID: <20081207193028.GA88077@fritz.cc.gt.atl.ga.us> On Sun, Dec 07, 2008 at 08:59:14AM +0100, Florian Weimer wrote: > >> How do you mean? Loke looking for a client that's making repeated dns > >> queries within the TTL? Maybe poorly coded bots? > And all UNIX-like clients which do not perform local caching by > default. 8-( I've found that linux hosts can be filtered by keeping state on AAAA lookups, followed by A lookups (glibc has this behavior). But with the growth of Vista, and other OS that perform AAAA lookups first, this filtering step is less and less useful. Subsequent checks for wpad resolutions can help, since vista revived this proxy discovery technique (for a period in a dangerous manner). Of course these heuristics have a limited shelf life. In a statistical sense, the failure to negatively cache or the failure to observe TTLs is a useful filter, but the results must then be correlated with other data (NS data sets, SLB data sets, perhaps p0f-style data sets) to remove FPs. I suggest the tool should consider keeping a window of state (e.g., AAAA --> A lookups, etc.), so that this filtering does not have to happen offline/post alert. Of the unappealing aspects of snort is that its alerts are too packet-specific (yes, there are flow bits, and simple ways to accumulate state). Here are some additional notes: -- if the tool is configured to reconstruct DNS sessions, you should handle DNS TC truncation bit packets; simply filtering per-packet allows for both IP and TC fragmentation. -- Consider filters for RFC suggested limits on NS counts (7 I believe). This is a good way to spot flux. Here is a distribution of NS counts for the top million or so (mostly whitelistable) domains (log scale count x NS count): http://fritz.cc.gt.atl.ga.us/ns_count_top1M_domains.txt (Note the drop at 12 NS; this is related to why we have only 13 root servers--512 byte payloads are difficult to pack with more labels, even after line compression.) The administrative burden of large numbers of NS is large, and the number of sites that exceed this for legitimate reasons is vanishingly small. Often, flux networks use >7 NS. This threshold, multiplied by AS diversity counts, is a very strong filter currently (FPs are only those global CDNs like akamai, which can be white listed). See the work of Thorsten Holz for early insights. I realize by suggesting this, flux networks can "reduce and evade". But this would be a good development since such networks are more brittle, and the lowered NS count makes them sensitive to diurnal patterns in victims. (Evolution can also weaken a species, and applying pressure to force flux conformance with usual DNS deployment patterns will diminish their capabilities.) This goes after a central weakness of flux: professional DNS is usually deployed on high availability systems--several 9s. But flux networks uses victim resources (ranked, but often poor in comparison to normal DNS resources), and handing authority to a non-reachable host results in failure signaling and possible botnet fragmentation. There are other layers of heuristics; hopefully these ideas let you design an architecture where one can express them in a simple syntax. -- David Dagon /"\ "When cryptography dagon at cc.gatech.edu \ / ASCII RIBBON CAMPAIGN is outlawed, bayl Ph.D. Student X AGAINST HTML MAIL bhgynjf jvyy unir Georgia Inst. of Tech. / \ cevinpl." From fw at deneb.enyo.de Sun Dec 7 15:21:53 2008 From: fw at deneb.enyo.de (Florian Weimer) Date: Sun, 07 Dec 2008 21:21:53 +0100 Subject: [Discussion] features (mainly dns) In-Reply-To: <20081207193028.GA88077@fritz.cc.gt.atl.ga.us> (David Dagon's message of "Sun, 7 Dec 2008 14:30:28 -0500") References: <1226959498.27961.3.camel@arodgers-panasonic> <4922E0C6.1090204@jonkmans.com> <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> <87y6ys4drh.fsf@mid.deneb.enyo.de> <20081207193028.GA88077@fritz.cc.gt.atl.ga.us> Message-ID: <87ljuru466.fsf@mid.deneb.enyo.de> * David Dagon: > On Sun, Dec 07, 2008 at 08:59:14AM +0100, Florian Weimer wrote: > >> >> How do you mean? Loke looking for a client that's making repeated dns >> >> queries within the TTL? Maybe poorly coded bots? > > > >> And all UNIX-like clients which do not perform local caching by >> default. 8-( > > I've found that linux hosts can be filtered by keeping state on AAAA > lookups, followed by A lookups (glibc has this behavior). Nice idea. But this depends on the application (it has to use getaddrinfo) and system configuration (GNU libc must detect some form of IPv6 connectivity). > -- Consider filters for RFC suggested limits on NS counts (7 I > believe). I don't think such a limit exists at the RFC level. There are certain TLD-specific limits. As you've mentioned, these are often bypassed my serving more NS records in the authority section of responses. You may also need to exclude (e)TLDs and several DNSBLs. Multiple A records per NS record are likely suspicious, too (but I think you've already mentioned that). From dagon at cc.gatech.edu Sun Dec 7 16:28:45 2008 From: dagon at cc.gatech.edu (David Dagon) Date: Sun, 7 Dec 2008 16:28:45 -0500 Subject: [Discussion] features (mainly dns) In-Reply-To: <87ljuru466.fsf@mid.deneb.enyo.de> References: <1226959498.27961.3.camel@arodgers-panasonic> <4922E0C6.1090204@jonkmans.com> <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> <87y6ys4drh.fsf@mid.deneb.enyo.de> <20081207193028.GA88077@fritz.cc.gt.atl.ga.us> <87ljuru466.fsf@mid.deneb.enyo.de> Message-ID: <20081207212845.GA88742@fritz.cc.gt.atl.ga.us> On Sun, Dec 07, 2008 at 09:21:53PM +0100, Florian Weimer wrote: > > I've found that linux hosts can be filtered by keeping state on AAAA > > lookups, followed by A lookups (glibc has this behavior). > > Nice idea. But this depends on the application (it has to use > getaddrinfo) and system configuration (GNU libc must detect some form > of IPv6 connectivity). I seem to recall lo0 having ::1/128 was enough to convince glibc that it should perform AAAA lookups; even if an answer is returned, glibc only later observes the 4-only interface needs an A record. (I would not consider this a bug per se, as I once did; it's just aggressive forward compliance.) This behavior is most acute in networks (e.g., a few cyber cafes) doing DNS filtering/typo mining, and dropping all AAAA traffic. So even though the network only hands out ipv4 addresses, the linux users experience timeouts for many common applications. The XP users just laugh. (I think OSX is well behaved, prioritizing A over AAAA?) But you're correct; there are some linux applications will not signal this way. (I think someone needs to take on the enormous project of stub/application fingerprinting and analysis. Particularly now that DNS prefetching is in Chrome an allegedly coming to firefox some day.) > > -- Consider filters for RFC suggested limits on NS counts (7 I > > believe). > > I don't think such a limit exists at the RFC level. Correct; rfc 1912 was non-standards track. It summarizes general rules of thumb for DNS deployment (most still relevant); 'violations' of those standards are useful first-order filtering rules. To Matt and the others in the project: I think in summary I can suggest three categories of DNS module filtering, based on the amount of time they require: -- rules that do inline per-packet checks (e.g., queries from DNSBL listed hosts; other IP-layer intel); this ideally works at line speed. Some narrow state window, on the order a small multiple of DNS timeouts, should let one do simple checks across DNS conversations. (E.g., a class of SLBs can be found by noting a non-response to AAAA?, and an affirmative response to A?.) Such rules are IPS candidates, garden-wall capable, etc. -- rules that can be done inline, but should also be done on subsequent updates of data feeds. (E.g., RRsets can be checked for dom and ip4 BL matches; subsequent updates should be *rechecked* so one can alert that a host was likely an early infection victim). This works at line speed; a lower priority thread rechecks key records based on updated intelligence feeds, DNSBLs, etc. You might expect DNSBLs will update once every 15-20 minutes; a suitably small time-based LRU structure may let you recheck a window of previously seen DNS traffic. Similarly, this class of includes rules that would trigger only after active traffic probes complete. (E.g., after witnessing a cautionary RRset, the tool queries for MX, SOA, and triggers only after analyzing the answer.) Assuming one finds heuristics that work and are not DoS contributors, these rules would need some time--multiples of average RTT times--to complete. So a short window of old traffic is useful; something more than flow bits. -- rules that require keeping complex state on a DNS conversation, and are therefore much slower. (E.g., delaying analysis of a TC truncated DNS packet, since we later observe the affected host is successfully doing a tcp connection, or edns0 is being used, etc. Counts of NS and other metrics are best done against the larger data). Machine learning fits here, potentially, but really might be off a span or using dedicated hosts. I think everything in this thread, even if not yet a mature stand-alone filter, can fall into one of three categories. (Is there a 4th?) This might help you design general properties of the dns detection module. A user would potentially label which rules fire at which stage of decoding: per-packet, per-DNS-conversation, or post-hoc analysis. We're entering an era where DNS vendors are now building IDS-style security into their offerings; even IPS capabilities. So perhaps the third category (certainly more difficult to code) is a lower priority in your tool. I almost think there's need to create a DNS-specific offering, since traffic aggregation for DNS clusters makes it easy to gather large amounts of traffic. The more traditional tcp and DPI work seems better done elsewhere in the edge. Perhaps your dns module, when run exclusively, would fill that role. -- David Dagon /"\ "When cryptography dagon at cc.gatech.edu \ / ASCII RIBBON CAMPAIGN is outlawed, bayl Ph.D. Student X AGAINST HTML MAIL bhgynjf jvyy unir Georgia Inst. of Tech. / \ cevinpl." From mcholste at gmail.com Mon Dec 8 19:57:34 2008 From: mcholste at gmail.com (Martin Holste) Date: Mon, 8 Dec 2008 18:57:34 -0600 Subject: [Discussion] Features - Designing for Scaleability In-Reply-To: <49370AFA.5020404@jonkmans.com> References: <49370AFA.5020404@jonkmans.com> Message-ID: Essentially, yes, but I think that "protocol" might sound a bit more daunting than the task really would be. I think "method" may be a more appropriate term. A simple web server that responds to HTTP posts would be good enough. My implementation uses the XMLRPC::Lite Perl module, which works perfectly to serialize and deserialize native Perl structures, making the actual transport completely transparent from hub to spoke. On Wed, Dec 3, 2008 at 4:40 PM, Matt Jonkman wrote: > So you're thinking the tool should have it's own management protocol to > a central hub built in. To allow sensors to be relatively autonomous, > pulling config and reporting data through a single protocol? > > Matt > > Martin Holste wrote: > > John, > > > > I think you've hit on some key points here. I manage about a dozen > > sensors, and there is a definite line that is crossed somewhere around a > > half-dozen servers where managing the configuration and health of the > > boxes requires some power tools. I prefer Nagios for general system > > health monitoring, so that has never been an issue. Conversely, it has > > been an incredible challenge to manage rules and granular Snort > > performance on many boxes with multiple Snort instances. I've taken the > > route you have with many home-grown scripts, etc., but there's always > > something new that comes out with Snort making the management > > difficult. Specifically, dealing with SO rules throws a huge wrench in > > the works when dealing with mixed architectures. > > > > My strategy thus far has been to have my sensors completely self > > contained and manage them centrally using a home-grown system written in > > Perl to make generic queries and issue commands. The databases which > > Snort writes to are on the box (performance really isn't impacted at all > > by doing this). The Perl agent receives queries like "get all alerts in > > last X minutes," or "add this rule to the running configuration," or > > "calculate stats for Mbps and alerts per second for last 1000 ticks." > > The key is that since all the data is on the sensor, it scales very > > well. The central Perl client can then parallelize queries so all > > sensors can search at the same time much faster than if all the alerts > > were located in one central database. Since everything goes through the > > central management client, it can easily log all the actions (and even > > queries) which are run for audit/change management purposes. > > > > For encryption, I run everything through OpenVPN. This works really > > well, especially for debugging, since it is much easier to tcpdump a > > tunnel interface than get a hook into a TLS socket. > > > > I'm working on a 2.0 version of this architecture which will be entirely > > asynchronous, so that the sensors can alert the central management hub > > on predetermined criteria. > > > > For truly large deployments, I think you're right that a parent-child > > setup might be necessary. An exploration of peer-to-peer techniques > > might be an interesting, but I think that for simplicity, a tree > > hierarchy would make the most sense. That is, a "hub" may have X number > > of servers assigned to it, and a hub higher up the tree would be able to > > ask hubs to delegate queries down the hierarchy. It would be > > interesting to see if there would be performance gains versus attempting > > parallelized queries over thousands of sensors from just one hub. My > > thinking is that some of the sensors could also function as hubs. > > > > I think that we need to remember that the vast majority of users do not > > deploy more than a few sensors, so we need to guard against spending too > > much devel time on features that will only serve a small percentage of > > the community. That said, audit, change management, archiving, and > > other management features are things that benefit everyone. As long as > > we keep it all modular, users can mix and match to get the features they > > require. > > > > --Martin > > > > On Tue, Nov 18, 2008 at 12:47 PM, John Pritchard > > > wrote: > > > > Team, > > > > My apologies if I've missed this being covered previously. Or, if the > > Bro framework already takes some of these things into consideration. > > > > I'd like to suggest that we take very large deployments into > > consideration when designing our solution. The kind of problems you > > encounter when managing an infrastructure with a handful or even a > > dozen different IDS sensors is very different than trying to > > efficiently and consistently manage infrastructures with larger > > footprints (e.g. > 100+ sensors). > > > > A couple of suggestions to help our design address these potential > > deployment and management scenarios: > > > > 1) Centralized sensor and policy management platform (or framework) > > --> Such a solution may be restricted to a single centralized server > > or multiple servers. > > --> Might be a parent -> child relationship among configuration > > servers to segregate business units, or simply replication among > > servers for disaster recovery / business continuity purposes > > --> efficient, repeatable, and audit-able methodology for making > > changes to both individual sensors as well as pre-defined groups of > > sensors (e.g. dmz sensors, dns sensors, development lab sensors, > > etc...) > > --> My experience to date has been performing these kind of tasks > with > > home-grown scripts, ssh, scp, audit logs, etc... However, it would be > > nice to find something a little more mature for this project. > > > > I have not used it, but there is a project called "Puppet" that looks > > like it might be a good candidate for trying to develop a framework > > along these lines: > > http://reductivelabs.com/trac/puppet/wiki/PuppetIntroduction > > > > > > 2) Centralized sensor and policy monitoring platform (or framework) > > --> This is similar to the "management framework" concept, but the > > focus is more on device health monitoring... and possibly policy > > integrity monitoring... > > --> For this piece, I'm thinking of something that provides functions > > such as looking at memory utilization, cpu utilization, hard-drive > > space, network interface stats... and other "bits" such as dates and > > checksums for critical configuration files changed (e.g. detection > > policies, tuning policies, variable definitions, etc)... > > > > There are a number of open-source enterprise monitoring utilities out > > there. Here's one I've been playing with recently: > > http://www.zabbix.com/ > > > > > > 3) Distributed Data Repositories > > I know we briefly touched on the database stuff when talking about > > schema design. I just wanted to add a plug for a couple of things > > here: > > --> encrypted communication channels (sensor -> DB or sensor -> > sensor) > > --> ability to simultaneously log to 2 or more data repositories > > > > I strongly agree with the concept of designing modular solutions. So, > > these kind of features can be "bolted on" if they were required. > > > > Look forward to everyone's thoughts on how we can most effectively > > tackle problems of scale for large deployments. > > > > Cheers, John > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > > > > > > > ------------------------------------------------------------------------ > > > > _______________________________________________ > > Discussion mailing list > > Discussion at openinfosecfoundation.org > > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081208/74bba3e0/attachment.html From jpleger at gmail.com Mon Dec 8 22:07:46 2008 From: jpleger at gmail.com (James Pleger) Date: Mon, 8 Dec 2008 20:07:46 -0700 Subject: [Discussion] Features - Designing for Scaleability In-Reply-To: References: <49370AFA.5020404@jonkmans.com> Message-ID: <221481660812081907jce385fbh6e3605441712bf6e@mail.gmail.com> This is a bit off topic, but I think that the way that argus client/server works might be a good place to start looking at some features. It would be awesome to run a client on my workstation and start feeding alerts with a particular ruleset. Speaking of rulesets, I can imagine having a daemon that listens for commands from a management console(clients) that tells it to load a ruleset and what to do with that data. This type of architecture could also work particularly well with trunked interfaces or different zones of trust, that different ids/ips engineers can manage(think of VRFs and PIX security contexts). It would be very neat to be able to start up new instances of ids detection modules on the fly with memory constraints and different settings(such as bpf filters, memory constraints, processor limits etc.) I hope my thoughts aren't too scattered. James Pleger e: jpleger at gmail.com On Mon, Dec 8, 2008 at 5:57 PM, Martin Holste wrote: > Essentially, yes, but I think that "protocol" might sound a bit more > daunting than the task really would be. I think "method" may be a more > appropriate term. A simple web server that responds to HTTP posts would be > good enough. My implementation uses the XMLRPC::Lite Perl module, which > works perfectly to serialize and deserialize native Perl structures, making > the actual transport completely transparent from hub to spoke. > > On Wed, Dec 3, 2008 at 4:40 PM, Matt Jonkman wrote: >> >> So you're thinking the tool should have it's own management protocol to >> a central hub built in. To allow sensors to be relatively autonomous, >> pulling config and reporting data through a single protocol? >> >> Matt >> >> Martin Holste wrote: >> > John, >> > >> > I think you've hit on some key points here. I manage about a dozen >> > sensors, and there is a definite line that is crossed somewhere around a >> > half-dozen servers where managing the configuration and health of the >> > boxes requires some power tools. I prefer Nagios for general system >> > health monitoring, so that has never been an issue. Conversely, it has >> > been an incredible challenge to manage rules and granular Snort >> > performance on many boxes with multiple Snort instances. I've taken the >> > route you have with many home-grown scripts, etc., but there's always >> > something new that comes out with Snort making the management >> > difficult. Specifically, dealing with SO rules throws a huge wrench in >> > the works when dealing with mixed architectures. >> > >> > My strategy thus far has been to have my sensors completely self >> > contained and manage them centrally using a home-grown system written in >> > Perl to make generic queries and issue commands. The databases which >> > Snort writes to are on the box (performance really isn't impacted at all >> > by doing this). The Perl agent receives queries like "get all alerts in >> > last X minutes," or "add this rule to the running configuration," or >> > "calculate stats for Mbps and alerts per second for last 1000 ticks." >> > The key is that since all the data is on the sensor, it scales very >> > well. The central Perl client can then parallelize queries so all >> > sensors can search at the same time much faster than if all the alerts >> > were located in one central database. Since everything goes through the >> > central management client, it can easily log all the actions (and even >> > queries) which are run for audit/change management purposes. >> > >> > For encryption, I run everything through OpenVPN. This works really >> > well, especially for debugging, since it is much easier to tcpdump a >> > tunnel interface than get a hook into a TLS socket. >> > >> > I'm working on a 2.0 version of this architecture which will be entirely >> > asynchronous, so that the sensors can alert the central management hub >> > on predetermined criteria. >> > >> > For truly large deployments, I think you're right that a parent-child >> > setup might be necessary. An exploration of peer-to-peer techniques >> > might be an interesting, but I think that for simplicity, a tree >> > hierarchy would make the most sense. That is, a "hub" may have X number >> > of servers assigned to it, and a hub higher up the tree would be able to >> > ask hubs to delegate queries down the hierarchy. It would be >> > interesting to see if there would be performance gains versus attempting >> > parallelized queries over thousands of sensors from just one hub. My >> > thinking is that some of the sensors could also function as hubs. >> > >> > I think that we need to remember that the vast majority of users do not >> > deploy more than a few sensors, so we need to guard against spending too >> > much devel time on features that will only serve a small percentage of >> > the community. That said, audit, change management, archiving, and >> > other management features are things that benefit everyone. As long as >> > we keep it all modular, users can mix and match to get the features they >> > require. >> > >> > --Martin >> > >> > On Tue, Nov 18, 2008 at 12:47 PM, John Pritchard >> > > wrote: >> > >> > Team, >> > >> > My apologies if I've missed this being covered previously. Or, if >> > the >> > Bro framework already takes some of these things into consideration. >> > >> > I'd like to suggest that we take very large deployments into >> > consideration when designing our solution. The kind of problems you >> > encounter when managing an infrastructure with a handful or even a >> > dozen different IDS sensors is very different than trying to >> > efficiently and consistently manage infrastructures with larger >> > footprints (e.g. > 100+ sensors). >> > >> > A couple of suggestions to help our design address these potential >> > deployment and management scenarios: >> > >> > 1) Centralized sensor and policy management platform (or framework) >> > --> Such a solution may be restricted to a single centralized server >> > or multiple servers. >> > --> Might be a parent -> child relationship among configuration >> > servers to segregate business units, or simply replication among >> > servers for disaster recovery / business continuity purposes >> > --> efficient, repeatable, and audit-able methodology for making >> > changes to both individual sensors as well as pre-defined groups of >> > sensors (e.g. dmz sensors, dns sensors, development lab sensors, >> > etc...) >> > --> My experience to date has been performing these kind of tasks >> > with >> > home-grown scripts, ssh, scp, audit logs, etc... However, it would >> > be >> > nice to find something a little more mature for this project. >> > >> > I have not used it, but there is a project called "Puppet" that >> > looks >> > like it might be a good candidate for trying to develop a framework >> > along these lines: >> > http://reductivelabs.com/trac/puppet/wiki/PuppetIntroduction >> > >> > >> > 2) Centralized sensor and policy monitoring platform (or framework) >> > --> This is similar to the "management framework" concept, but the >> > focus is more on device health monitoring... and possibly policy >> > integrity monitoring... >> > --> For this piece, I'm thinking of something that provides >> > functions >> > such as looking at memory utilization, cpu utilization, hard-drive >> > space, network interface stats... and other "bits" such as dates and >> > checksums for critical configuration files changed (e.g. detection >> > policies, tuning policies, variable definitions, etc)... >> > >> > There are a number of open-source enterprise monitoring utilities >> > out >> > there. Here's one I've been playing with recently: >> > http://www.zabbix.com/ >> > >> > >> > 3) Distributed Data Repositories >> > I know we briefly touched on the database stuff when talking about >> > schema design. I just wanted to add a plug for a couple of things >> > here: >> > --> encrypted communication channels (sensor -> DB or sensor -> >> > sensor) >> > --> ability to simultaneously log to 2 or more data repositories >> > >> > I strongly agree with the concept of designing modular solutions. >> > So, >> > these kind of features can be "bolted on" if they were required. >> > >> > Look forward to everyone's thoughts on how we can most effectively >> > tackle problems of scale for large deployments. >> > >> > Cheers, John >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> > >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> > >> > >> > >> > ------------------------------------------------------------------------ >> > >> > _______________________________________________ >> > Discussion mailing list >> > Discussion at openinfosecfoundation.org >> > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion >> >> -- >> -------------------------------------------- >> Matthew Jonkman >> Emerging Threats >> Phone 765-429-0398 >> Fax 312-264-0205 >> http://www.emergingthreats.net >> -------------------------------------------- >> >> PGP: http://www.jonkmans.com/mattjonkman.asc >> >> > > > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion > > From lists at inliniac.net Tue Dec 9 03:27:13 2008 From: lists at inliniac.net (Victor Julien) Date: Tue, 09 Dec 2008 09:27:13 +0100 Subject: [Discussion] features (mainly dns) In-Reply-To: <87ljuru466.fsf@mid.deneb.enyo.de> References: <1226959498.27961.3.camel@arodgers-panasonic> <4922E0C6.1090204@jonkmans.com> <20081206124309.GA76697@fritz.cc.gt.atl.ga.us> <87y6ys4drh.fsf@mid.deneb.enyo.de> <20081207193028.GA88077@fritz.cc.gt.atl.ga.us> <87ljuru466.fsf@mid.deneb.enyo.de> Message-ID: <493E2BE1.7090607@inliniac.net> Florian Weimer wrote: > * David Dagon: > >> On Sun, Dec 07, 2008 at 08:59:14AM +0100, Florian Weimer wrote: >> >>>>> How do you mean? Loke looking for a client that's making repeated dns >>>>> queries within the TTL? Maybe poorly coded bots? >> >> >>> And all UNIX-like clients which do not perform local caching by >>> default. 8-( >> I've found that linux hosts can be filtered by keeping state on AAAA >> lookups, followed by A lookups (glibc has this behavior). > > Nice idea. But this depends on the application (it has to use > getaddrinfo) and system configuration (GNU libc must detect some form > of IPv6 connectivity). > >> -- Consider filters for RFC suggested limits on NS counts (7 I >> believe). > > I don't think such a limit exists at the RFC level. There are certain > TLD-specific limits. As you've mentioned, these are often bypassed my > serving more NS records in the authority section of responses. You > may also need to exclude (e)TLDs and several DNSBLs. > > Multiple A records per NS record are likely suspicious, too (but I > think you've already mentioned that). I think the idea of having spamassassin-style scoring could be useful here. One suspicious (but rfc valid) property of a dns request could then be ignored, but if there is more suspiciousness than a certain threshold, alert. Regards, Victor From jonkman at jonkmans.com Thu Dec 18 16:10:33 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Thu, 18 Dec 2008 16:10:33 -0500 Subject: [Discussion] Rules Syntax Message-ID: <494ABC49.2040504@jonkmans.com> The first thing we'd like to do in technical terms is setup a rules syntax. IMHO we have to dump the existing ways that's done and start over with what we'll be doing for the next 10 years. We're trying to plan a meetup where we can brainstorm in person, maybe around RSA in california in a couple months. But I'd like to start a discussion now. So I think the best way to start to structure a syntax is to express what we want to say in english. So here's one: --- If someone thats not in a group I like and has a reputation that's not more bad than this in the category relevant to the type of traffic this is makes a full connection to me and does this particular bad thing then add a .5 to their local reputation, if they're over 1.5 then block them for 5 minutes. --- Other examples of what you'd like to be able to say, in english, and we'll start to find the similarities and begin to make a syntax. Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Dec 19 15:15:42 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 19 Dec 2008 15:15:42 -0500 Subject: [Discussion] OS Fingerprinting Message-ID: <494C00EE.5020802@jonkmans.com> Decula in IRC had two great ideas. One was to use something like p0f to do live OS fingerprinting. That could be very useful for eliminating false positives and identifying unusual behavior (ie a windows box running a telnet server, etc) Adding this to the wiki, anyone have thoughts to add to that? Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Fri Dec 19 15:17:04 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 19 Dec 2008 15:17:04 -0500 Subject: [Discussion] Hooks for Other than Blocking Message-ID: <494C0140.2010606@jonkmans.com> The other decula idea was to have hooks in the engine for actions other than blocking, like redirecting/forwarding. I like this idea a lot as well. Snort bait n switch style, redirect an attacker to a honeypot. Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From scheidell at secnap.net Fri Dec 19 15:17:55 2008 From: scheidell at secnap.net (Michael Scheidell) Date: Fri, 19 Dec 2008 15:17:55 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: <494C00EE.5020802@jonkmans.com> References: <494C00EE.5020802@jonkmans.com> Message-ID: <494C0173.6010909@secnap.net> Matt Jonkman wrote: > Decula in IRC had two great ideas. One was to use something like p0f to > do live OS fingerprinting. > > That could be very useful for eliminating false positives and > identifying unusual behavior (ie a windows box running a telnet server, etc) > > Adding this to the wiki, anyone have thoughts to add to that? > > Matt > p0f can't tell the difference between a windows XP workstation and windows 2000 server (last I remember). I had used it for 'zombie' detection in our anti-spam system, but the incremental assistance wasn't worth the cpu (it took a little cpu.) -- Michael Scheidell, CTO Phone: 561-999-5000, x 1259 > *| *SECNAP Network Security Corporation * Certified SNORT Integrator * King of Spam Filters, SC Magazine 2008 * Information Security Award 2008, Info Security Products Guide * CRN Magazine Top 40 Emerging Security Vendors _________________________________________________________________________ This email has been scanned and certified safe by SpammerTrap(r). For Information please see http://www.secnap.com/products/spammertrap/ _________________________________________________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20081219/98ef1976/attachment.html From jonkman at jonkmans.com Fri Dec 19 15:22:59 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Fri, 19 Dec 2008 15:22:59 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: <494C0173.6010909@secnap.net> References: <494C00EE.5020802@jonkmans.com> <494C0173.6010909@secnap.net> Message-ID: <494C02A3.2010302@jonkmans.com> I use it with spamassassin and it *seems* to make a big difference there. I haven't pulled exact stats, but the idea that mail from a windows box is more likely to be spam is surely valid. Have to look into how far it does ID windows versions. Even if we can just get server os vs workstation os that'd be pretty interesting I think. Anyone used it much lately? matt Michael Scheidell wrote: > > > Matt Jonkman wrote: >> Decula in IRC had two great ideas. One was to use something like p0f to >> do live OS fingerprinting. >> >> That could be very useful for eliminating false positives and >> identifying unusual behavior (ie a windows box running a telnet server, etc) >> >> Adding this to the wiki, anyone have thoughts to add to that? >> >> Matt >> > p0f can't tell the difference between a windows XP workstation and > windows 2000 server (last I remember). > I had used it for 'zombie' detection in our anti-spam system, but the > incremental assistance wasn't worth the cpu (it took a little cpu.) > > > -- > Michael Scheidell, CTO > Phone: 561-999-5000, x 1259 >> *| *SECNAP Network Security Corporation > > * Certified SNORT Integrator > * King of Spam Filters, SC Magazine 2008 > * Information Security Award 2008, Info Security Products Guide > * CRN Magazine Top 40 Emerging Security Vendors > > > ------------------------------------------------------------------------ > > This email has been scanned and certified safe by SpammerTrap?. > For Information please see www.secnap.com/products/spammertrap/ > > > ------------------------------------------------------------------------ > -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From ddpbsd at gmail.com Fri Dec 19 15:26:32 2008 From: ddpbsd at gmail.com (ddp) Date: Fri, 19 Dec 2008 15:26:32 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: <494C00EE.5020802@jonkmans.com> References: <494C00EE.5020802@jonkmans.com> Message-ID: On Fri, Dec 19, 2008 at 3:15 PM, Matt Jonkman wrote: > Decula in IRC had two great ideas. One was to use something like p0f to > do live OS fingerprinting. > > That could be very useful for eliminating false positives and > identifying unusual behavior (ie a windows box running a telnet server, etc) > > Adding this to the wiki, anyone have thoughts to add to that? > > Matt > > -- > -------------------------------------------- > Matthew Jonkman > Emerging Threats > Phone 765-429-0398 > Fax 312-264-0205 > http://www.emergingthreats.net > -------------------------------------------- > > PGP: http://www.jonkmans.com/mattjonkman.asc > > > Add in functionality like pads or passer.py and you're on your way to recreating Sourcefire's RNA. passer.py (http://stearns.org/passer) does OS identification also. dan From dagon at cc.gatech.edu Fri Dec 19 15:38:09 2008 From: dagon at cc.gatech.edu (David Dagon) Date: Fri, 19 Dec 2008 15:38:09 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: <494C02A3.2010302@jonkmans.com> References: <494C00EE.5020802@jonkmans.com> <494C0173.6010909@secnap.net> <494C02A3.2010302@jonkmans.com> Message-ID: <20081219203809.GA11449@fritz.cc.gt.atl.ga.us> On Fri, Dec 19, 2008 at 03:22:59PM -0500, Matt Jonkman wrote: > I use it with spamassassin and it *seems* to make a big difference > there. I haven't pulled exact stats, but the idea that mail from a > windows box is more likely to be spam is surely valid. > > Have to look into how far it does ID windows versions. Even if we can > just get server os vs workstation os that'd be pretty interesting I think. > > Anyone used it much lately? Since this thread is "for the whiteboard", I'll describe the pony I'd like for xmas: -- your tool should allow me to hook a p0f module, or my own DSO that performs immediate classifications. (These classifications could then trigger more than logging; e.g., firewall-like match rules, e.g., "drop if win95", etc.) The callback is immediate. -- the tool should also allow me to hook a DSO that does active probing. p0f does not catch them all, and so I might want to initiate some active probes of an IP witnessed in flows (e.g., some pen testers jiggle 137/139/445 to get a version string). Let's put aside the dos-enabling potential for a moment, for purposes of this example. Say instead I might want to consult a database about the flow pairs, and wait out a SELECT, or a dnsbl rtt. Whatever; after my probes/additional inquiry complete, I may have further classifications to report, and more firewall behaviors to trigger. Here, the callback is not immediate, but assync. I.e., there are quick-and-dirty OS fingerprinting techniques that one can use via a pluggable module. There are also some active measurements or correlations that can do a better job of fingerprinting--allow these as well. These would be invoked at the operator's own risk. But if you permit an async update on flow classifications, you will create the API that permits new innovations, instead of merely integrating existing opensource technologies. So that's my pony; hopefully others want it as well. -- David Dagon /"\ "When cryptography dagon at cc.gatech.edu \ / ASCII RIBBON CAMPAIGN is outlawed, bayl Ph.D. Student X AGAINST HTML MAIL bhgynjf jvyy unir Georgia Inst. of Tech. / \ cevinpl." From pepperjack at autoshun.org Fri Dec 19 16:20:14 2008 From: pepperjack at autoshun.org (Jack Pepper) Date: Fri, 19 Dec 2008 15:20:14 -0600 Subject: [Discussion] OS Fingerprinting In-Reply-To: <494C02A3.2010302@jonkmans.com> References: <494C00EE.5020802@jonkmans.com> <494C0173.6010909@secnap.net> <494C02A3.2010302@jonkmans.com> Message-ID: <20081219152014.0rhygdzb400gkk40@mail.afferentsecurity.com> Quoting Matt Jonkman : > I use it with spamassassin and it *seems* to make a big difference > there. I haven't pulled exact stats, but the idea that mail from a > windows box is more likely to be spam is surely valid. > > Have to look into how far it does ID windows versions. Even if we can > just get server os vs workstation os that'd be pretty interesting I think. > > Anyone used it much lately? I still use it at the "Genre" level and it works predictably. I categorize things into "Windows", "Linux", "Unix" (aix+sun), "BSD", and "MAC" and it seems to work well enough ( ~ 80% ? ) to feed data into my "poor-man's RNA". I don't think the fine grained accuracy is reliable for service-patch level detection, but p0f works ok at the genre level. I did rewrite the socket listener and caching part of it, but the fingerprinting part works well enough as-is. tc -- Simple compliance is a hacker's best friend ---------------------------------------------------------------- @fferent Security Labs: Isolate/Insulate/Innovate http://www.afferentsecurity.com From thorsten.holz at informatik.uni-mannheim.de Fri Dec 19 17:20:42 2008 From: thorsten.holz at informatik.uni-mannheim.de (Thorsten Holz) Date: Fri, 19 Dec 2008 23:20:42 +0100 Subject: [Discussion] Hooks for Other than Blocking In-Reply-To: <494C0140.2010606@jonkmans.com> References: <494C0140.2010606@jonkmans.com> Message-ID: On 19.12.2008, at 21:17, Matt Jonkman wrote: > I like this idea a lot as well. Snort bait n switch style, redirect an > attacker to a honeypot. I like that idea, too! Could be interesting to divert traffic based on certain characteristics. We played with bait 'n switch in the past and could use it for several honeypot setups. Cheers, Thorsten From c.criscione at securenetwork.it Sat Dec 20 06:16:53 2008 From: c.criscione at securenetwork.it (Claudio Criscione) Date: Sat, 20 Dec 2008 12:16:53 +0100 Subject: [Discussion] Hooks for Other than Blocking In-Reply-To: References: <494C0140.2010606@jonkmans.com> Message-ID: <200812201216.53429.c.criscione@securenetwork.it> On Friday 19 December 2008 23:20:42 Thorsten Holz wrote: > On 19.12.2008, at 21:17, Matt Jonkman wrote: > > I like this idea a lot as well. Snort bait n switch style, redirect an > > attacker to a honeypot. Redirection could also be used to escalate to more CPU intensive checks (antiviruses?), or to provide human feedbacks in order to do some supervised learning. Think about blocking some "high confidence" attacks and introducing some human interaction on more uncertain results in order to improve detection with time. -- Claudio Criscione Secure Network S.r.l Via Venezia, 23 - 20099 Sesto San Giovanni (MI) - Italia Tel: +39 02.24126788 Mob: +39 392 3389178 email: c.criscione at securenetwork.it web: www.securenetwork.it From c.criscione at securenetwork.it Sat Dec 20 06:25:19 2008 From: c.criscione at securenetwork.it (Claudio Criscione) Date: Sat, 20 Dec 2008 12:25:19 +0100 Subject: [Discussion] Rules Syntax In-Reply-To: <494ABC49.2040504@jonkmans.com> References: <494ABC49.2040504@jonkmans.com> Message-ID: <200812201225.19575.c.criscione@securenetwork.it> On Thursday 18 December 2008 22:10:33 Matt Jonkman wrote: > The first thing we'd like to do in technical terms is setup a rules > syntax. IMHO we have to dump the existing ways that's done and start > over with what we'll be doing for the next 10 years. [...] > Other examples of what you'd like to be able to say, in english, and > we'll start to find the similarities and begin to make a syntax. Well, what about: "if someone in my organization has never started any ftp traffic in the last three months starts an ftp connection, notify me and start watching more carefully that person. " - Someone vs some machine Using the IP address is still the only way to go in most cases, but we need more sophisticate means to identify who's who as networks evolve (think about whole cities behind a NAT) - In my organization vs in my internal lan Networks are melting into clouds (i like the cloud vaporware ;-)), with VPNs and such, and I'm not even talking about virtualization (more about that inf uture posts). - in the last three monts can actually be translated to "is not used to" or "does not usually" The issue with statistical approaches is that you really have to develope custom models. What about "signature based statistical models"? - "watching more carefully" I'm not sure we always want the same "resolution" on network traffic, and I feel it would be great to be able to zoom on suspicious activity automatically without having to carry the burden of logging everything everytime Just my 2c :) -- Claudio Criscione From jonkman at jonkmans.com Sun Dec 21 13:11:02 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 21 Dec 2008 13:11:02 -0500 Subject: [Discussion] Rules Syntax In-Reply-To: <200812201225.19575.c.criscione@securenetwork.it> References: <494ABC49.2040504@jonkmans.com> <200812201225.19575.c.criscione@securenetwork.it> Message-ID: <494E86B6.8030509@jonkmans.com> Claudio Criscione wrote: > > Well, what about: > > "if someone in my organization has never started any ftp traffic in the last > three months starts an ftp connection, notify me and start watching more > carefully that person. " I like this too. How do we store that kind of data for long term? Even if we were to just store last timestamp we saw that port in use on this IP that'd be a significant amount of data on the average net to go back months, no? How do we go after that then? (Call to the db experts out there) Use a sliding scale so as the user-defined storage space allocated fills up older data drops out? Use a limited range of ports, and/or group together high port ranges? > > - Someone vs some machine > Using the IP address is still the only way to go in most cases, but we need > more sophisticate means to identify who's who as networks evolve (think about > whole cities behind a NAT) I think we should think more inside the firewall for these issues, no? There are ways, and several commercial products that track a user to an IP in realtime. Cisco I believe does, and others surely. LDAP integration/AD, netbios login monitoring, etc. It's possible, but it's a big thing to tackle. And likely we'd have patent conflicts. We can explore that though if there's a large enough driver to get it. Thoughts? > - in the last three monts can actually be translated to "is not used to" > or "does not usually" > The issue with statistical approaches is that you really have to develope > custom models. What about "signature based statistical models"? Yes, statistical approaches are tough. I'd like to see what is available out there in this area these days as far as open research. As I mentioned, I think it'll be a good use of some of our grant money to contract or grant fund a real statistician or group of such. Maybe we could get it made into a class project at a university somewhere under the guidance of an experienced statistician. > > - "watching more carefully" > I'm not sure we always want the same "resolution" on network traffic, and I > feel it would be great to be able to zoom on suspicious activity > automatically without having to carry the burden of logging everything > everytime > Another good point. Most folks these days do that with rotating tcpdumps, but you're time limited there. If you don't get to that alert before the pcap rotates out you've lost it. Are there better approaches out there? Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Dec 21 13:13:07 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 21 Dec 2008 13:13:07 -0500 Subject: [Discussion] Hooks for Other than Blocking In-Reply-To: <200812201216.53429.c.criscione@securenetwork.it> References: <494C0140.2010606@jonkmans.com> <200812201216.53429.c.criscione@securenetwork.it> Message-ID: <494E8733.4040801@jonkmans.com> Claudio Criscione wrote: > Redirection could also be used to escalate to more CPU intensive checks > (antiviruses?), or to provide human feedbacks in order to do some supervised > learning. I like that idea. Use circumstances to help decide if a binary needs to be quarantined/av scanned. Maybe source, have we seen god/bad binaries from this source before, size of the binary (haven't seen many 50meg viruses of late), etc. What other factors might we consider? > Think about blocking some "high confidence" attacks and introducing some > human interaction on more uncertain results in order to improve detection > with time. What kind of human interaction do you mean here? Human approval? Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Dec 21 13:22:20 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 21 Dec 2008 13:22:20 -0500 Subject: [Discussion] Hooks for Other than Blocking In-Reply-To: References: <494C0140.2010606@jonkmans.com> Message-ID: <494E895C.504@jonkmans.com> What if we were to integrate something like nepenthes into this? As a side module,. Instead of redirecting traffic to some other place, why nit just ingest it and let nepenthes generate faked responses? So for instance we see a highly reliable netbios attack from an internal host to another internal host. We then take all future traffic from the attacker and push it into the nepenthes module for say 60 seconds. If nepenthes reports real exploitation then we generate big red flashing lights and continue to interfere with/drop all future traffic from that attacker. If nepenthes does not report exploitation we turn the redirect off and let the alleged attacker go on their way, hopefully with only a minor burp in network connectivity. Matt Thorsten Holz wrote: > On 19.12.2008, at 21:17, Matt Jonkman wrote: > >> I like this idea a lot as well. Snort bait n switch style, redirect an >> attacker to a honeypot. > > I like that idea, too! Could be interesting to divert traffic based on > certain characteristics. We played with bait 'n switch in the past and > could use it for several honeypot setups. > > Cheers, > Thorsten > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Dec 21 13:29:57 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 21 Dec 2008 13:29:57 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: <20081219203809.GA11449@fritz.cc.gt.atl.ga.us> References: <494C00EE.5020802@jonkmans.com> <494C0173.6010909@secnap.net> <494C02A3.2010302@jonkmans.com> <20081219203809.GA11449@fritz.cc.gt.atl.ga.us> Message-ID: <494E8B25.9030909@jonkmans.com> David Dagon wrote: > > Since this thread is "for the whiteboard", I'll describe the pony I'd > like for xmas: > > -- your tool should allow me to hook a p0f module, or my own DSO that > performs immediate classifications. (These classifications could > then trigger more than logging; e.g., firewall-like match rules, > e.g., "drop if win95", etc.) > > The callback is immediate. > I like this a lot. I am very anxious to have a rule syntax option of something like p0f:checkif,iswin95. Keeping an internal db of known OS's shouldn't be tough. And it'd be very interesting to get alerts when an OS changes. > -- the tool should also allow me to hook a DSO that does active > probing. p0f does not catch them all, and so I might want to > initiate some active probes of an IP witnessed in flows (e.g., > some pen testers jiggle 137/139/445 to get a version string). Dangers there, but interesting. We'd have to make sure the actual knowledge of what OS's are being seen is valuable enough for the overhead of going to see what they are. But well worth it. I think PADS has been mentioned in the thread already. The tech exists to do so for sure. > > Let's put aside the dos-enabling potential for a moment, for > purposes of this example. Say instead I might want to consult a > database about the flow pairs, and wait out a SELECT, or a dnsbl > rtt. Whatever; after my probes/additional inquiry complete, I may > have further classifications to report, and more firewall > behaviors to trigger. > > Here, the callback is not immediate, but assync. > > I.e., there are quick-and-dirty OS fingerprinting techniques that one > can use via a pluggable module. There are also some active > measurements or correlations that can do a better job of > fingerprinting--allow these as well. These would be invoked at the > operator's own risk. > > But if you permit an async update on flow classifications, you will > create the API that permits new innovations, instead of merely > integrating existing opensource technologies. > > So that's my pony; hopefully others want it as well. > I think you've got something here for sure. Summarizing this into the wiki. Which by the way, the current list is at: http://doc.emergingthreats.net/bin/view/Main/EngineFeatures Thanks all! Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From jonkman at jonkmans.com Sun Dec 21 13:33:15 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Sun, 21 Dec 2008 13:33:15 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: References: <494C00EE.5020802@jonkmans.com> Message-ID: <494E8BEB.2040204@jonkmans.com> ddp wrote: > Add in functionality like pads or passer.py and you're on your way to recreating > Sourcefire's RNA. passer.py (http://stearns.org/passer) does OS > identification also. I'll look this one over, thanks! As for recreating RNA, we definitely don't want to do that. 1. It's been done, and apparently works well 2. It's been done. i.e. patented. :) We're definitely doing different things though, and I think we'll have the opportunity to add a good deal more to it, and make better use of the info in a new rules language. Matt -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc From lists at inliniac.net Mon Dec 22 08:02:59 2008 From: lists at inliniac.net (Victor Julien) Date: Mon, 22 Dec 2008 14:02:59 +0100 Subject: [Discussion] OS Fingerprinting In-Reply-To: <20081219152014.0rhygdzb400gkk40@mail.afferentsecurity.com> References: <494C00EE.5020802@jonkmans.com> <494C0173.6010909@secnap.net> <494C02A3.2010302@jonkmans.com> <20081219152014.0rhygdzb400gkk40@mail.afferentsecurity.com> Message-ID: <494F9003.6040507@inliniac.net> Jack Pepper wrote: > Quoting Matt Jonkman : > >> I use it with spamassassin and it *seems* to make a big difference >> there. I haven't pulled exact stats, but the idea that mail from a >> windows box is more likely to be spam is surely valid. >> >> Have to look into how far it does ID windows versions. Even if we can >> just get server os vs workstation os that'd be pretty interesting I think. >> >> Anyone used it much lately? > > I still use it at the "Genre" level and it works predictably. I > categorize things into "Windows", "Linux", "Unix" (aix+sun), "BSD", > and "MAC" and it seems to work well enough ( ~ 80% ? ) to feed data > into my "poor-man's RNA". > > I don't think the fine grained accuracy is reliable for service-patch > level detection, but p0f works ok at the genre level. I did rewrite > the socket listener and caching part of it, but the fingerprinting > part works well enough as-is. I wonder if it's useful to look at stuff like User-Agent headers and HTTP server signatures as well to determine the OS. It's easy to spoof of course, but p0f isn't perfect either. Could be interesting for cases where p0f and other methods give different results. Shouldn't happen all that often in normal traffic. (although I have one example in my own daily usage, the Webmail plugin for Mozilla Thunderbird that I use to read hotmail uses a Mac OS X user-agent on my Ubuntu box) Cheers, Victor From jonkman at jonkmans.com Mon Dec 22 10:00:17 2008 From: jonkman at jonkmans.com (Matt Jonkman) Date: Mon, 22 Dec 2008 10:00:17 -0500 Subject: [Discussion] OS Fingerprinting In-Reply-To: <494F9003.6040507@inliniac.net> References: <494C00EE.5020802@jonkmans.com> <494C0173.6010909@secnap.net> <494C02A3.2010302@jonkmans.com> <20081219152014.0rhygdzb400gkk40@mail.afferentsecurity.com> <494F9003.6040507@inliniac.net> Message-ID: <494FAB81.1090906@jonkmans.com> Interesting idea, to compare user-agent strings to the p0f output. In normal user workstations an altered UA is very suspicious. An alert saying that the workstation just sent a linux or mac UA would be good. Or even if it changed language. A lot of malware that runs through the sandnet uses a normal looking windows UA, but it's russian. Or they'll use an Opera string with a russian tag. But overall, just the fact that the US has changed is very interesting, especially in a user net. Matt Victor Julien wrote: > Jack Pepper wrote: >> Quoting Matt Jonkman : >> >>> I use it with spamassassin and it *seems* to make a big difference >>> there. I haven't pulled exact stats, but the idea that mail from a >>> windows box is more likely to be spam is surely valid. >>> >>> Have to look into how far it does ID windows versions. Even if we can >>> just get server os vs workstation os that'd be pretty interesting I think. >>> >>> Anyone used it much lately? >> I still use it at the "Genre" level and it works predictably. I >> categorize things into "Windows", "Linux", "Unix" (aix+sun), "BSD", >> and "MAC" and it seems to work well enough ( ~ 80% ? ) to feed data >> into my "poor-man's RNA". >> >> I don't think the fine grained accuracy is reliable for service-patch >> level detection, but p0f works ok at the genre level. I did rewrite >> the socket listener and caching part of it, but the fingerprinting >> part works well enough as-is. > > I wonder if it's useful to look at stuff like User-Agent headers and > HTTP server signatures as well to determine the OS. It's easy to spoof > of course, but p0f isn't perfect either. > > Could be interesting for cases where p0f and other methods give > different results. Shouldn't happen all that often in normal traffic. > > (although I have one example in my own daily usage, the Webmail plugin > for Mozilla Thunderbird that I use to read hotmail uses a Mac OS X > user-agent on my Ubuntu box) > > Cheers, > Victor > _______________________________________________ > Discussion mailing list > Discussion at openinfosecfoundation.org > http://lists.openinfosecfoundation.org/mailman/listinfo/discussion -- -------------------------------------------- Matthew Jonkman Emerging Threats Phone 765-429-0398 Fax 312-264-0205 http://www.emergingthreats.net -------------------------------------------- PGP: http://www.jonkmans.com/mattjonkman.asc