[Oisf-users] Announce: StreamDB

[Martin Holste]

I wanted to let these communities know about a new open-source project
called StreamDB (http://code.google.com/p/streamdb/) I've just
published that's proven to be extremely helpful for my analysts.  It
is a fast and simple tool for quickly viewing traffic related to IDS
alerts (or any IP-based event) which specializes in ultra-fast
retrievals from very large data sets.  It can hook into Snorby as it
is OpenFPC compatible.  It is also very effective for PCRE searching
traffic from a given source or destination IP address.  Streams are
rotated out based on configured retention size in a ring-buffer
fashion.  From the project home page:

StreamDB is a high-performance framework for storing network streams.
The current version uses Vortex IDS to read the streams from a file or
network interface and saves them to an indexed DB and data file. Web
code provides an URL-based query interface. There is also a
command-line interface which includes the ability to read piped
queries from STDIN. In addition to almost instant retrieval by IP
address, StreamDB also allows PCRE searches and file type searches on
streams if an IP address is provided as an initial filter. The system
can handle recording gigabit line-speed networks and can retrieve
arbitrary streams from terabytes of data in milliseconds. It is
designed to be a complimentary tool to intrusion detection systems to
aid security analysts.

Here are some query examples:

http://streamdb/?srcip=10.0.0.1
http://streamdb/?srcip=10.0.0.1&dstip=1.1.1.1&dstport!80
http://streamdb/?srcip=10.0.0.1&dstip=1.1.1.1&dstport=80&start=2 weeks
ago&end=now
http://streamdb/?srcip=10.0.0.1&pcre=example.com
http://streamdb/?srcip=10.0.0.1&pcre=MZ.*PE\x00\x00
http://streamdb/?srcip=10.0.0.1&sort=1&as_hex=1
http://streamdb/?srcip=10.0.0.1&raw=1
http://streamdb/?srcip=10.0.0.1&offset=1000&limit=200
http://streamdb/?srcip=10.0.0.1&filetype=executable

Examples from the CLI:
./sdb --srcip 10.0.0.1 --filetype pdf --headers-only
tail -f /var/log/snort/alert | ./sdb > alert_streams.txt

All of these queries will return in a second or two, assuming that the
IP's referred to aren't busy web servers or NAT points.  IP's with
many connections will benefit from more specific search filters for
time and/or destination IP address.  Some non-scientific benchmarks on
commodity hardware searching 5 TB of data: PCRE search for a given
srcip with 1000 connections completes in about one second.  A similar
search for a srcip with 50,000 connections will take about two
minutes.  A lookup for a given srcip/dstip pair will complete in less
than a second, including browser render time.

Example output as text/plain:

Returning 2 of 2 at offset 0 from Sun Jan 30 11:56:11 2011 to Sun Jan
30 11:56:11 2011

2011-01-30 11:56:11 192.168.58.52:4099 -> 131.243.2.191:80 13s 512
bytes FIN ASCII text, with CRLF line terminators

GET /bro-workshop-2009-2/slides/Installation.pdf HTTP/1.1
Host: www.bro-ids.org
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13)
Gecko/20101206 Ubuntu/10.10 (maverick) Firefox/3.6.13
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Referer: http://www.bro-ids.org/bro-workshop-2009-2/slides/
X-Do-Not-Track: 1
Connection: keep-alive



2011-01-30 11:56:11 192.168.58.52:4099 <- 131.243.2.191:80 13s 778247
bytes FIN PDF document, version 1.3

200 OK
Connection: Keep-Alive
Date: Sun, 30 Jan 2011 17:56:11 GMT
Accept-Ranges: bytes
ETag: "8f724e-bde6b-47617252eebc0"
Server: Apache/2.2.17 (FreeBSD) mod_ssl/2.2.17 OpenSSL/0.9.8e DAV/2
PHP/5.3.5 with Suhosin-Patch mod_python/3.3.1 Python/2.6.6
mod_wsgi/2.8
Content-Length: 777835
Content-Type: application/pdf
Last-Modified: Sat, 17 Oct 2009 01:08:07 GMT
Keep-Alive: timeout=5, max=100
X-HTTP-Version: 1.1

%PDF-1.3
%...........
4 0 obj
<< /Length 5 0 R /Filter /FlateDecode >>
stream
x..VM..7..W.T..z,i4....I.-.bS..!.a...nl..=. ...%O.Q.....|.Y.(..|.......).
9........l.h.'E....-....&.7]....... .}Fvr....}.x...
.)...^k."U.rC.....w.N<...Z..u<..Z..e.j....4T.Hpj.........u...../g.....n.....o.......R....*.Do.9<.*]>...(...I8....ikJ_.T...:......c|..Ki..Q..>.U.MZ...*...!........jKik~7.qg.iw#.......|..............D/.\Yx..v...2<..d.	O./...<...&.DDF....x..K.
Qy...|..f	F>.2k.v....^v..{^<..Q..v....t}.f;x.e.S..]...U.7......l..uD...
<clipped for brevity>

I hope that others have found it as useful as we have for rapidly
investigating IDS alerts.  Please use the project page or email me to
let me know about any questions, issues, or suggestions.

Thanks,

Martin