[Oisf-devel] OISF State of the Project Report Phase Two

Wed Mar 2 17:06:45 UTC 2011

We are embarking on our Phase Two objectives for Suricata. We've just had a great brainstorming session at RSA in San Francisco and a lot of great ideas came of it.

Getting the Phase Two development Roadmap set for Suricata was our primary goal of the meeting, but we also had a good deal of normal and administrative foundation business to discuss. I'll summarize some of the things discussed and our solutions if possible. This document is intended to spur further discussion and solicit input on all administrative items as well as the Phase Two Dev Roadmap. So please share your thoughts. We'd like to keep discussion centered as either comments to our blog post at http://www.openinfosecfoundation.org, or as email discussions on the OISF Discussion mailing list at http://lists.openinfosecfoundation.org/mailman/listinfo/discussion.

Administrative Items;

1. Two Board of Directors seats are up for re-election. Myself and Jennifer Steffens are stepping down from the board. Nominations are open through March 4th. Elections will begin Monday March 7th and will be announced on the website and lists.

2. We need to solidify a Bug disclosure policy and publish. Suggested was to ask disclosers for a 4 week period to fix and test the bug before disclosure for critical bugs. Exceptions will be possible. Commits to the git repository will be disclosure essentially so we do not intend to keep issues private longer than necessary for affected users and vendors to pull and patch.

3. We need to get our documentation better organized and more complete. Anne-Fleur Koolstra is our technical writer and she is working quickly toward this goal. 

4. The foundation's funding this year is less than last year. It has been suggested that we use some of our funding to hire an individual to seek and write grants. We are going to seek an individual to do so as we know there are many grants we are eligible for but we haven't the resources/expertise to pursue them. If you know a good grant writer please send him/her our way!

Phase Two Development Roadmap

Here are the items that were tentatively included as targets for Phase Two. These are only tentative, we will have to assess resources required for each item and plan accordingly.

High Priority Items
1. CUDA GPU Acceleration
Work is ongoing and major hurdles have been overcome. This still has some issues to solve but appears feasible.

2. IP and DNS Reputation
We have three major items to build here. Matching already exists in the engine, but we will need to design:
a. a local hub
b. a hub to sensor communication protocol
c. a master distribution hub
d. a master hub to local hub communication protocol

It has been suggested that we separate this out to a separate project within the OISF. I think this would be a great idea if we can find separate funding and expertise. Will seek that, but for the short term we'll try to execute using existing resources. We should also consider using the Collective Intelligence Sharing Framework recently released.

3. File Extraction and Inspection
This has been independently funded and work is underway. We will have at least four new keywords: fileext:, filetype:, filename:, and filestore. This will use the UNIX "file" utility's magic bytes database allowing Suricata to identify several thousand files by name and magic bytes. Comparison can be made of type by magic bytes to extension reported and extract the file from the stream to save in a directory with metadata for other processing.

4. UNIX Socket Output
Relatively simple to execute. Will allow alerts to be fed to a socket for pickup by outside processes.

5. Rotating Pcap Support
A patch exists to implement and will be incorporated.

6. IP Only Match Payload Capture
Trivial to implement. 

7. SCADA Preprocessors
Digital Bond has graciously given us the go ahead to port their SCADA preprocessors to Suricata. We will attempt this in Phase Two if resources permit. 

8. Replace Keyword
WIth limits this is relatively easy to implement. 

9. GeoIP Keyword
Using databases such as Maxmind to be able to query geo information about an IP. Relatively easy to implement.

10. Performance Stats Additions
In process

11. DNS Fast Flux and Anomaly Preprocessor
Build a table of dns lookups by source and result. This will allow near time analysis to find fast flux domain names, hosts that do more lookups than their peers, and hosts that to more lookups that result in low ttl, etc. 

12. Mysql/Postgress/Sguil Output
Support for a direct to database output. Not best for performance, but required by many users. 

13. FTP Data Stream Port Prediction
Track ftp control sessions and mark data streams to allow them to be ignored or focused upon in rules.

14. Max Inspection Time Cutoff
Allow a packet to be passed after x time even if not processed to control latency.

15. Stateful Pattern Matching/Transaction-Aware Detections
Adding additional application awareness for pattern matching. 

16. Live Ruleset Swap
Allow ruleset updates without losing flow state.

Lower Priority Items

1. Global Shared Flowvars
Requires more research into impact.

2. Built in Rule Testing
Tabled for future development

4. Netflow/IPFIX Output
Ability to send netflow data to collectors, allowing the sensor to assist with netflow collection. Also allow output of alert data via ipfix for more direct transmission to correlation devices. Tabled for future resources.

5. Host/App/OS Table import
To allow reassembly decisions, but more importantly to allow querying within a rule what the target or source is to eliminate false positives. the OpenDPI framework has been suggested as a method to import. Seeking resources to make this happen.

6. Snortsam Output Plugin
Tabled to be either replaced by IP Reputation, or resources become available to implement.

7. Regex optimization
Continuing search for an implementation.

8. Live Engine Status Page
Tabled till resources become available.

9. Additional Protocol Recognition
May be solved by an API for host/app import similar to DPI. Reference Qosmos.

10. Using Entropy to Identify Hostile Binaries
There has been some research here and the idea was suggested in conjunction with file extraction. Tabled till extraction is effective and then we can experiment with entropy.

Please let us know your thoughts, and let me know if I missed anything that was discussed. We will put out a final Phase Two Development Roadmap within a couple of weeks. Coding is already underway on existing feature items.

----------------------------------------------------
Matthew Jonkman
Emergingthreats.net
Emerging Threats Pro
Open Information Security Foundation (OISF)
Phone 765-807-8630
Fax 312-264-0205
http://www.emergingthreatspro.com
http://www.openinfosecfoundation.org
----------------------------------------------------

PGP: http://www.jonkmans.com/mattjonkman.asc