[Discussion] OISF Brainstorming Session Summary / Phase Three Draft Dev Roadmap

Sat Sep 24 12:13:42 UTC 2011

http://www.openinfosecfoundation.org/OISF_Phase_Three_Dev_Roadmap_v1.pdf

Brainstorming Session Summary
Phase Three Dev Roadmap
Developed at RAID 2011, Menlo Park, San Francisco CA
Monday September 19, 2011

The Open Information Security Foundation (OISF) is a non-profit foundation organized to build a next generation IDS/IPS engine, Suricata. 

The primary goal of this our fourth Brainstorming Session was to review and adjust the Development Roadmap for Phase Three. Following is the result of that meeting and discussion.

Topics and features that gained consensus and are reasonably feasible are listed here. They are rated by priority and resources required. Highest priority items with the least resources required will generally be the first to be attempted. This is a work in progress. All comments, additions, adjustments welcome! 

First, overall results of the meeting:

Bro
The Bro team was present and extremely helpful, thanks to all! We learned a lot about our similarities and differences, and have identified a number of places where code could be shared, event data and even reputation data. We are resolved to pursue a much closer relationship with the Bro team and Bro itself, including exploring how Suricata and Bro can work together in realtime to share data and events. They are very complementary tools. 

Remote Attendance
We made our first attempt at mass remote attendance using a conference line, Google+ hangouts, and then Webex. The conference line was not useful. Google+ was extremely useful but we unfortunately hit the max attendees in the room quickly. (This max has since been lifted by Google). We then shifted to Webex which was a great success. 

Remote users had sound difficulties which we can solve technically at the next meeting. But were able to speak to the room via PA and see via video. Feedback was extremely positive. We will plan to make this a more technically solid feature at the next brainstorming meeting. We will also shoot for a more neutral time of day to accommodate more eastern and western hemisphere users. 

Library-ization
It was agreed to continue to build and develop toward being able to separate Suricata’s core functions to libraries to make them available for use in other tools. Lib-HTP is already a separate module. Suricata’s stream reassembly will be the next function to be separated into a library over time. 

Universal Ruleset Language
The idea of a universal descriptive language was brought up for discussion again. There are a number of organizations that have expressed interest in this idea, so we will pursue getting a project started. This could be the foundation for a future new language for Suricata. 

The concept in brief is a universal and very descriptive language that can be used to describe the exploitation of vulnerabilities, traffic on a network, state of a protocol, etc. The goal being to have a central language where an issue can be described and output parsers that put that into many different formats, including snort syntax, blacklists, url blocklists, other IDS formats, web app firewalls, etc. 

Roadmap Features:

SSL Analyzer: High Priority / Medium Resources required
This module will be implemented in two phases. The first phase will do the following:

Verify the cert validity, chain, and check CRL
Allow admins to white/blacklist certain certificates 
Log certificates used, cipher, and timestamp
User configurable alerts based on CA, whether a cert has been seen recently or ever, when a cert is bad or on a CRL. 

Phase two will include the ability to decrypt sessions where keys are intercept-able, and the ability to provide private keys for local ssl relationships for decryption and analysis. We will consider the use of commodity crypto acceleration cards for this phase especially considering their reasonable cost.

IP and DNS Reputation Distribution: High Priority / High Resources Required
IP reputation is already implemented in the Suricata Engine, but only at a basic level. We have stumbled on development of a larger system to allow live bi-directional data distribution due to the scale of the project and numerous issues to solve. 

It was decided by the group to take this in smaller steps, and thus this phase will take an incremental step forward and design a static database model that will be extremely small (less than 10megabytes) that could be imported on a regular basis and replace in -memory data on demand. This will allow immediate use of IP Reputation data and allow live updates with minimal transport complexity. 

Configuration options will be made available to set thresholds per category to allow either alerting or blocking per stream at setup, and checks within the rules language to allow queries. A directive such as this will be the goal:

ip_rep:<category>,<=>,int;    

This will return true or false. The user should be able to determine if a no-data means true or false. 

We also need to look into using something like the Common Intelligence Framework (CIF), and other similar projects for transport of data.

DNS Preprocessor and Anomaly Detection: High Priority / Medium Resources
This preprocessor will process and log all dns requests detected in udp or tcp. Logging will be for a user-configurable time and into an in-memory database of a format to be determined. Query and response and attributes will be logged. Alerting may be done based on user-configurable attributes.

This module should be prepared to interface with DNS Reputation services to be implemented later and be able to make an alert/block decision based on blacklisting of the domain requested. Perhaps even fake the response to a defined IP and block the true response.

A second portino of this task will build an external tool (i.e. not in the packet processing path) to analyze this DNS database for at least the following attributes:

Hosts with significantly more frequent lookups than peers in their network.
Hosts with lookups resulting in frequent low TTL responses
Domains that resolve to different IP addresses frequently
Possible analysis of variance in DNS queries for the same domain (potential covert channels)
Very regularly timed queries for the same domain name.

This module should make available the rules directive dns_name: parameter, which will be the contents of the domain name in a DNS query, or if known the domain name that resulted in the response containing a specified IP. For example, to check that the remote IP address in a rule for a Microsoft Patch update came in the response to a DNS query containing microsoft.com. 

Time Based Counters: Medium Priority / Medium Resources Required
This will be a flowbit/flowvar or thresholding style directive to allow checking of timestamp of a previous event. For example, to set a flowbit and timestamp for an event, upon a second occurrence verify that it was within or outside a certain time interval. A command and Control channel for example that checkins in every 15 minutes but uses a url that is not extremely unique. If it happens on a regular basis be able to create an event. 

FTP Port Prediction: Low Priority / Low Resources
This module would be an addition to the current ftp protocol identification module. It will watch the port commands on the control channels to predict the data port coming. Then allow analysis of just that data stream.

GEO IP: High Priority / Low Resources
This module will use a geo-ip database such as Maxmind to allow geolocation of IP addresses. 

Live Ruleset Swapping: High Priority / Medium Resources
This will allow signaling Suricata to begin a ruleset reload while still running. It will reload from the same configuration files and then swap to the new ruleset using the same state table and var table. 

This will of course require more memory in many cases, but was determined to be a sufficiently valuable feature. 

Thanks to everyone who was able to attend in person and remotely. Turnout was great and the discussions as always were very productive and collaborative. Suricata is a great tool because of the community, it’s collective intelligence, and the willingness of everyone involved to chip in and help. I know I speak for the entire OISF team in saying thank you for your energy and ideas.

----------------------------------------------------
Matt Jonkman
Emerging Threats Pro
Open Information Security Foundation (OISF)
Phone 866-504-2523 x110
http://www.emergingthreatspro.com
http://www.openinfosecfoundation.org
----------------------------------------------------

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4364 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/discussion/attachments/20110924/6520f292/attachment.bin>