[Oisf-users] Pcap file carving (was: Quickstart for Bro Cluster)

[Kevin Ross]

Still a work in progress & playing with it. I have found having sigs to
automatically move over PDFs with JavaScript, Embedded Files etc & even
converting the PEID sigs to clamav to move over (known) packed files helps
too. I would have preferred to use NFEX (
http://blogs.cisco.com/security/network-based-file-carving/) but that
doesn't seem to like multiple PCAPs being fed into it, I would have
preferred that as that is more session based.

My aim isn't to capture & store all EXEs (well not the now, I was thinking
perhaps depending on storage they would be stored and keep rescanning them
for a few days to see if anything new pops up) but more to get across either
the known malicious ones and the suspicious ones. So far seems to be working
pretty well and at least now if I see an alert for JavaScript for a PDF or
something I can go to the PDF and see what that JavaScript was and so on so
it is a big help. Even the performance is surprisingly good on standard
hardware. I am also going to put out Dionaea honeyports (
http://dionaea.carnivore.it/) to capture stuff in the environment being
spread.

It may not be perfect and it won't get everything (it will miss malicious
EXEs I am sure either through lack of knowledge to highlight it as a risk or
obfuscated & custom packed exes) but good for a free solution. As I said the
Mal. Analyst's Cookbook has been a good help for ideas, tools and things to
extend it (automatic analysis scripts, automatic unpacking examples scripts
etc) and for ideas. Hopefully as time goes on and more stuff goes into it
the less will slip past without being kept for me to look at. Results can be
useful anyway and has found malware. In doing this I have found the hardest
parts to do was the carving and then more importantly the classification for
what should be sent for further analysis (later I might have a better way to
further filter out ok binaries out of the ones flagged by clamav.
Considering this is coming off the network and picking out stuff it is
pretty cool.

/mal_binaries/00000001.pdf: PDF Javascript Function Declared.UNOFFICIAL
FOUND
/mal_binaries/00000001.pdf.001: PDF Embedded EXE MZ PE.UNOFFICIAL FOUND
/mal_binaries/00000002.exe: PACKER.PEnguinCryptv10.UNOFFICIAL FOUND
/mal_binaries/00000001.exe: PACKER.UPXv20MarkusLaszloReiser.UNOFFICIAL
FOUND

Regards, Kevin


On 29 September 2011 20:24, Martin Holste <mcholste at gmail.com> wrote:

> > 2) I then have a script which runs every 4 mins and calls tcpxtract to
> > directly carve the files out of those PCAPs (currently EXEs & PDFs) and
> then
> > once that is done clamav is called with some additional sigs to help
> > highlight suspicious things too
>
> That's a really mature setup!  I don't do anything with ClamAV right
> now.  I used to auto-submit to VirusTotal, but their non-web interface
> was taking around two hours for a scan result, so it was usually not
> tactically helpful.
>
> I will plug my streamdb.googlecode.com project as a great way of
> bypassing daemonlogger and tcpxtract.  What I do is use a script which
> receives a notification every time a POLICY EXE sig hits with the
> src/dst IP, which is all streamdb needs to instantly grab the stream
> and carve the executable on-the-fly.  You can even retroactively
> search by any libmagic label (filetype=executable) or PCRE
> (pcre=PE\x00\x00) so you can use it to find PDF's, DOC, Flash, etc.
> Each extracted object has an object ID (oid), so as long as the
> streams are around, you have an indexed ID for that exact object
> observed in the network.  That makes attaching it to tickets or
> emailing it to other analysts trivial, and you know they'll be working
> on the same file you were.
>
> I thought this was worth mentioning streamdb again because I just
> updated the download tarball to include my committed code fixes which
> take care of the issues MySQL 5.1 was having; it should now work with
> any MySQL version.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20110930/b4e17e1f/attachment-0002.html>