[Oisf-devel] Extracting file from stream
Josh
josh at securemind.org
Thu Jan 21 19:57:48 EST 2010
I've been doing the stream extraction, and file rebuilding for AV testing for a
while and have managed to build up a fairly decent list of identifiers for
tcpxtract. If anyone wants them here goes:
tcpxtract.conf
----
avi(4000000, RIFF\?\?\?\?);
mpg(4000000, \x00\x00\x01\xba, \x00\x00\x01\xb9);
mpg(4000000, \x00\x00\x01\xb3, \x00\x00\x01\xb7);
fws(4000000, FWS);
art(150000, \x4a\x47\x04\x0e, \xcf\xc7\xcb);
art(150000, \x4a\x47\x03\x0e, \xd0\xcb\x00\x00);
gif(3000000, \x47\x49\x46\x38\x37\x61, \x00\x3b);
gif(3000000, \x47\x49\x46\x38\x39\x61, \x00\x00\x3b);
jpg(1000000, \xff\xd8\xff\xe0\x00\x10, \xff\xd9);
jpg(1000000, \xff\xd8\xff\xe1);
png(1000000, \x50\x4e\x47\?, \xff\xfc\xfd\xfe);
bmp(100000, BM\?\?\x00\x00\x00);
tif(200000000, \x49\x49\x2a\x00);
doc(12500000, \xd0\xcf\x11\xe0\xa1\xb1);
pst(400000000, \x21\x42\x4e\xa5\x6f\xb5\xa6);
ost(400000000, \x21\x42\x44\x4e);
dbx(4000000, \xcf\xad\x12\xfe\xc5\xfd\x74\x6f);
idx(4000000, \x4a\x4d\x46\x39);
mbx(4000000, \x4a\x4d\x46\x36);
html(50000, \x3chtml, \x3c\x2fhtml\x3e);
pdf(5000000, \x25PDF, \x25EOF\x0d);
mail(500000, \x41\x4f\x4c\x56\x4d);
ra(1000000, \x2e\x72\x61\xfd);
ra(1000000, \x2eRMF);
zip(10000000, PK\x03\x04, \x3c\xac);
java(1000000, \xca\xfe\xba\xbe);
gzip(1000000, \x1f\x8b\x08);
script(1000000, \x23\x21\x2f);
ne(40000000, \x4D\x5A\x50, \x4E\x45);
elf(30000000, \x7F\x45\x4C\x46);
exe(10000000, \x4d\x5a);
pe(10000000, \x50\x45\x00\x00);
ne(40000000, \x4D\x5A\x50, \x4E\x45);
#rar(10000000, Rar!
\x1A\x07\x00\x90\x73\x00\x00\x0d\x00\x00\x00\x00\x00\x00\x00);
mp3(4000000, ID3);
mp4(4000000, \x00\x00\x00\x18\x66\x74\x79\x70);
odt(12500000, \x50\x4b\x03\x04\x14);
torrent(10000000, \x64\x38\x3aannounce);
tgp(1000000, \x00\x00\x00\?\x66\x74\x79\x70\x33\x67\x70\x34);
pdf1(5000000, \x25PDF, \x25EOF\x0d);
pdf2(5000000, \x25PDF, \x25EOF\x0a);
pdf3(5000000, \x25PDF, \x25EOF\x0d\x0a);
torrent1(10000000, \x64\x38\x3a\x61\x6e\x6e\x6f\x75\x6e\x63\x65, \x65\x65);
rar1(10000000, Rar\x21);
svg(100000000, \x53 \x56 \x47 \x20 \x47 \x72 \x61 \x70 \x68 \x69 \x63 \x73
\x20 \x46 \x69 \x6C \x65 \x00);
doc(100000000, \x0D \x44 \x4F \x43);
docx(\x0D \x44 \x4F \x43);
log(100000000, \x4C \x6F \x67 \x66 \x69 \x6C \x65 \x20 \x6F \x66 \x20 \x54
\x72 \x65 \x6E \x64 \x20 \x4D \x69 \x63 \x72 \x6F \x20 \x48 \x69 \x6A \x61
\x63 \x6B \x54 \x68 \x69 \x73);
msg(1000000000, \xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00);
pages(10000000, \x50 \x4B \x03 \x04);
rtf(1000000000, \x7B \x5C \x72 \x74 \x66);
txt(1000000000, \x2A \x2A \x2A \x2A \x2A \x2A \x2A \x2A \x20 \x45 \x61 \x73
\x79 \x50 \x6C \x6F \x74 \x20 \x73 \x61 \x76 \x65 \x20 \x66 \x69 \x6C \x65
\x20 \x2A \x2A \x2A \x2A \x2A \x2A \x2A \x2A \x0D \x0);
wpd(1000000000, \xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00);
wps(1000000000, \xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00);
123(1000000000, ??? )need hex file
accdb(10000000, \x00 \x01 \x00 \x00 \x53 \x74 \x61 \x6E \x64 \x61 \x72 \x64
\x20 \x41 \x43 \x45 \x20 \x44 \x42);
csv(???) hex unknown
dat(1000000000, \x73 \x6C \x68 \x21);
db(10000000000, \xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x3E \x00 \x03
\x00 \xFE \xFF \x09 \x00 \x06 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00
x\x00 \x00);
dll(100000000, \x4D \x5A);
mdb(100000000, \x00 \x01 \x00 \x00 \x53 \x74 \x61 \x6E \x64 \x61 \x72 \x64
\x20 \x4A \x65 \x74 \x20 \x44 \x42 \x00 );
pps(100000000, hex unknown);
ppt(100000000,\xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00 \x00 \x00 \x00);
pptx(10000000, \x50 \x4B \x03 \x04 );
sdb(100000000, \x6 \x00 \x00 \x73 \x64 \x62 \x66 \x02 \x78 \xEF \xCD \xAB \x89
\x20 \x06 );
vcf(100000000, \x42 \x45 \x47 \x49 \x4E \x3A);
wks(100000000, \xFF \x00 \x02 \x00 \x04 \x04 \x05 \x54 \x02 \x00);
xls(100000000, \xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00 );
xlsx(10000000, \x50 \x4B \x03 \x04 );
xml(100000000, \x FF \xFE \x3C \x00 \x3F \x00 \x78 \x00 \x6D \x00 \x6C \x00
\x20 \x00 \x76 \x00 \x65 \x00 \x72 \x00 \x73 \x00 \x69 \x00 \x6F \x00 \x6E
\x00 \x3D \x0);
pct(1000000000, hex unknow);
bmp(1000000000, \x41 \x57 \x42 \x4D );
gif(1000000000, \x47 \x49 \x46 \x38 );
jpg(1000000000, \xFF \xD8 \xFF);
png(1000000000, \x89 \x50 \x4E \x47 \x0D \x0A \x1A \x0A \x00 \x00 \x00 \x0D
\x49 \x48 \x44 \x52);
psd(1000000000, \x38 \x42 \x50 \x53 \x00 \x01 \x00 \x00 \x00 \x00
\x00 \x00 \x00);
psp(1000000000, \x7E \x42 \x4B \x00);
thm(1000000000, \xFF \xD8 \xFF \xE1 \x07 \xFC \x45 \x78 \x69 \x66 \x00 \x00
\x49 \x49 \x2A \x00 \x08 \x00 \x00 \x00 \x09 \x00 \x0F \x01 \x02 \x00 \x06
\x00 \x00 \x00 \x7A \x00 \x00 \x00 \x10 \x01 \x02 \x00 \x13 \x00 \x00 \x00
\x80 \x00 \x00 \x00 \x12 \x01 \x03 \x00 \x01 \x00 \x00 \x00);
tif(100000000, \x49 \x49 \x2A \x00 );
ai(1000000000, \x25 \x21 \x50 \x53 \x2D );
drw(100000000, \x0D \x43 \x41 \x44 \x53 \x20 \x44 \x52 \x57 \x20);
eps(100000000, /xC5 /xD0 /xD3 /xC6);
psd(100000000, \x38 \x42 \x50 \x53 \x00 \x01 \x00 \x00 \x00 \x00 \x00 \x00
\x00);
3dm(10000000, \x43 \x61 \x64 \x65 \x6E \x74 \x20 \x33 \x44 \x20 \x4D \x6F \x64
\x65 \x6C \x20 \x56);
dwg(10000000, /x41 /x43);
pln(10000000, \xEF \xBB \xBF \x3C \x3F \x78 \x6D \x6C \x20 \x76 \x65 \x72 \x73
\x69 \x6F \x6E \x3D \x22 \x31 \x2E \x30 \x22 \x20);
indd(1000000, \x06 \x06 \xED \xF5 \xD8 \x1D \x4 \xE5 \xBD \x31 \xEF \xE7 \xFE
\x74 \xB7 \x1D \x44 \x4F \x43 \x55 \x4D \x45 \x4E \x54);
pdf(10000000, \x25 \x50 \x44 \x46 \x2D \x31 \x2E );
qxd(10000000, \x 58 \x50 \x52 \x33 );
qxp(10000000, \x58 \x50 \x52);
iff(10000000, \x46 \x4F \x52 \x4D );
m3u(10000000, \x23 \x45 \x58 \x54 \x4D \x33 \x55 \x0D \x0A );
mid(10000000, \x4D \x54 \x68 \x64 \x00 \x00 \x00 \x06 \x00 \x01 \x00);
mp3(10000000, \x49 \x44 \x33 \x03 );
ra(100000000, \x2E \x52 \x4D \x46 \x00 \x00 \x00 \x12 \x00);
wav(100000000, \x52 \x49 \x46 \x46);
wma(100000000, \x30 \x26 \xB2 \x75 \x8E \x66 \xCF \x11 \xA6 \xD9 \x00 \xAA
\x00 \x62 \xCE \x6C );
3g2(100000000, \x66 \x74 \x79 \x70);
asf(1000000, \x66 \x74 \x79 \x70);
asx(1000000, \x3c);
avi(10000000, \x52 \x49 \x46 \x46);
flv(100000000, \x46 \x4C \x56 \x01);
mkv(100000000,\x1A \x45 \xDF \xA3 \x93 \x42 \x82 \x88 \x6D \x61 \x74 \x72 \x6F
\x73 \x6B \x61 \x42 \x87 \x81 \x01 \x42 \x85 \x81 \x01 \x18 \x53 \x80 \x67);
mov(100000000, \x6D);
mp4(100000000, \x00 \x00 \x00);
mpg(100000000, \x00 \x00 \x01 \xBA \x44);
rm(1000000000, \x2E \x52 \x4D \x46 \x00 \x00 \x00 \x12 \x00 );\
swf(100000000, x\46 \x57 \x53);
vob(10000000, \x00 \x00 \x01 \xBA \x44);
wmv(100000000, \x30 \x26 \xB2 \x75 \x8E \x66 \xCF \x11 \xA6 \xD9 \x00 \xAA
\x00 \x62 \xCE \x6C );
rss(1000000,\x3C \x3F \x78 \x6D \x6C \x20 \x76 \x65 \x72 \x73 \x69 \x6F \x6E
\x3D \x22 \x31 \x2E \x30 \x22);
fnt(10000000, \x45 \x6C \x65 \x63 \x62 \x79 \x74 \x65 \x46 \x6E \x74 \x00);
fon(10000000, \x4D \x5A );
otf(10000000, \x4F \x54 \x54 \x4F \x00);
ttf(10000000, \x00 \x01 \x00 \x00 \x00 );
cab(10000000,\x4D \x53 \x43 \x46 \x00 \x00 \x00 \x00 );
cpl(10000000, \xDC \xDC);
cur(10000000, \x00 \x00 \x02 \x00);
key(10000000, \x48 2B \x42 \x45 \x44 \x56 \x20 \x50 \x72 \x6F \x64 \x75 \x63
\x74 \x73 \x20 \x4C \x69 \x63 \x65 \x6E \x73 \x65 \x20 \x4B \x65 \x79 \x20
\x46 x\69 \x6C \x65 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x1A);
lnk(1000000, \x4C \x00 \x00 \x00 \x01 \x14 \x02 \x00 \x00 \x00 \x00 \x00 \xC0
\x00 \x00 \x00 \x00 \x00 \x00 \x46);
sys(10000000, \x50 \x53 \x32 \x44);
cgf(10000000, \x45 \x4 \x7 \x33 \x3 \x2 \x43 \x4 \xD \x50 \x5 \xF \x32);
prf(100000000, \x41 \x74 \x68 \x65 \x72 \x6F \x73 \x20 \x50 \x72 \x6F \x66
\x69 \x6C \x65 \x20 \x46 \x69 \x6C \x65);
app(100000000, \xB8 \x0B \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00);
com(1000000, \xE9 \x4A \x2C \x90 \x90 \xCD \xAB \x43 \x6F \x70 \x79 \x72 \x69
\x67 \x68 \x74 \x20 \x28 \x43 \x29);
exe(1000000, \x4D \x5A \x0A \x00 \x02 \x00 \x00 \x00 \x04 \x00 \x0F \x00 \xFF
\xFF \x00 \x00 \xC0 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x40 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 );
7z(10000000, \x37 \x7A \xBC \xAF \x27 \x1C);
deb(1000000, \x21 \x3C \x61 \x72 \x63 \x68 \x3E \x0A \x64 \x65 \x62 \x69 \x61
\x6E \x2D \x62 \x69 \x6E \x61 \x72 \x79 \x20 \x20 \x20);
gz(10000000, \x1F \x8B \x08);
pkg(1000000, \x23 \x20 \x50 \x61 \x43 \x6B \x41 \x67 \x45 \x20 \x44 \x61 \x54
\x61 \x53 \x74 \x52 \x65 \x41 \x6D \x0A);
rar(10000000, \x52 \x61 \x72 \x21 \x1A \x07 \x00);
sit(10000000, \x53 \x74 \x75 \x66 \x66 \x49 \x74 \x20 \x28 \x63 \x29 \x31 \x39
\x39 \x37 \x2D);
sitx(1000000, \x53 \x74 \x75 \x66 \x66 \x49 \x74 \x21);
zip(1000000, \x50 \x4B \x03 \x04);
bin(10000000, \x42 \x4C \x49 \x32 \x32 \x33 \x51);
hqx(10000000,\x28 \x54 \x68 \x69 \x73 \x20 \x66 \x69 \x6C \x65 \x20 \x6D \x75
\x73 \x74 \x20 \x62 \x65 \x20 \x63 \x6F \x6E \x76 \x65 \x72 \x74 \x65 \x64
\x20 \x77 \x69 \x74 \x68 \x20 \x42 \x69 \x6E \x48 \x65 \x78 \x20 );
tmp(1000000000, \x4C \x59 \x31 \x44 \x4C \x20 \x41 \x6E \x6F \x74 \x68 \x65
\x72 \x20 \x63 \x61 \x72 \x64 \x66 \x69 \x6C \x65 \x20 \x2D \x20 \x52 \x54
\x46);
dmg(100000000, \x78);
toast(10000000, \x45 \x52 \x02 \x00 \x00);
vcd(100000000, \x45 \x4E \x54 \x52 \x59 \x56 \x43 \x44 \x02 \x00 \x00 \x01
\x02 \x00 \x18 \x58);
nes(10000000, \x4E \x45 \x53 \x1A);
rom(10000000, \x41 \x42);
sav(10000000, \x3C \x43 \x61 \x75 \x6C \x64 \x72 \x6F \x6E \x20 \x43 \x68 \x75
\x6E \x6B \x20 \x46 \x69 \x6C \x65 \x3E \x01 \x00 \x00 \x00 \x00 \x00 \x01
\x00);
msi(10000000, \xD0 \xCF \x11 \xE0 \xA1 \xB1 \x1A \xE1 \x00 \x00 \x00 \x00 \x00
\x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x00 \x3E \x00 \x03 \x00
\xFE \xFF \x09 \x00 \x06);
torrent(10000000, \x64 \x38 \x3A \x61 \x6E \x6E \x6F \x75 \x6E \x63 \x65);
---
- Josh
On Thursday 21 January 2010 05:58:45 Edward Bjarte Fjellskål wrote:
> Yao-Min Chen wrote:
> > One reason for doing full capture and file extraction is to detect
> > malware files in transit, so we can either block the files or
> > immediately report the host that receives such a file. The latter can
> > be used as a trigger for first responses.
> >
> > If Suricata can do this in memory instead of handing off the pcap files
> > to external tools there is efficiency and response time to be gained.
> >
> > Yaomin
>
> Hi list,
>
> My aim with fpcgui (Full Packet Capture GUI) is for offloading this
> from an IDS sensor. The IDS could have a preprocessor (or just a tool
> that reads the unified log for extraction of just the sessions that
> trigger events (If you know Sourcefire, im planing to use Estreamer for
> this)) that sends off a command to fpcgui, that would carve out the pcap
> of the session in question.
>
> You can then have a ringbuffer of full pcap data, and yet another
> ringbuffer with pcaps from sessions that triggered events etc.
>
> For each new sessions that is automagically(tm) carved out, you can
> send that through (example) tcpxtract->clamav and if virus found,
> send an event to your favorite event monitoring system (Sguil etc.).
>
> Read my last blogpost of fpcgui if it sounds interesting.
> http://www.gamelinux.org/?p=67
>
> I have a demo up, so you can test it if someone is interested.
>
> e
>
> > On 01/20/10 23:38, Victor Julien wrote:
> >> The ISC post lists quite a few tools that already support extracting
> >> files from pcaps. Is there something new and unsupported by those tools
> >> you are looking for in Suricata?
> >>
> >> Will Metcalf wrote:
> >>> Jerry,
> >>>
> >>> We will keep this in mind, although I think stuff like this may belong
> >>> in post-analysis. That being said does anybody have an interest in
> >>> flow/full traffic capture as an option?
> >>>
> >>> Regards,
> >>>
> >>> Will
> >>>
> >>> On Wed, Jan 20, 2010 at 4:22 PM, Jerry <jerry at cybercave.cz
> >>> <mailto:jerry at cybercave.cz>> wrote:
> >>>
> >>> Hi development team/list,
> >>> I have a question regarding features development. Are you planning
> >>> to include extraction files from packet stream into Suricata?
> >>>
> >>> It would be nice to have something that covers this issue:
> >>> http://isc.sans.org/diary.html?storyid=6961
> >>>
> >>> Thank you very much in advance
> >>>
> >>> Jerry
> >>>
> >>> --
> >>> Defending network against intrusion is like trying to keep a squid
> >>> inside a mesh bag. Question is, who will give up first :)
> >>>
> >>> _______________________________________________
> >>> Oisf-devel mailing list
> >>> Oisf-devel at openinfosecfoundation.org
> >>> <mailto:Oisf-devel at openinfosecfoundation.org>
> >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >>>
> >>>
> >>>
> >>> -----------------------------------------------------------------------
> >>>-
> >>>
> >>> _______________________________________________
> >>> Oisf-devel mailing list
> >>> Oisf-devel at openinfosecfoundation.org
> >>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
> >
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Oisf-devel mailing list
> > Oisf-devel at openinfosecfoundation.org
> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
> _______________________________________________
> Oisf-devel mailing list
> Oisf-devel at openinfosecfoundation.org
> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>
More information about the Oisf-devel
mailing list