[Oisf-devel] file extraction -- Re: [COMMIT] OISF branch, master, updated. a556338936ad3cd2b0379a6985fb62084368d99e
Victor Julien
victor at inliniac.net
Tue Nov 29 14:49:27 UTC 2011
This push adds the initial support of the file extraction functionality.
It adds a new build dependency: magic.h (libmagic-dev)
See the included rules/files.rules file for example usage.
Feedback highly appreciated.
Cheers,
Victor
On 11/29/2011 03:44 PM, noreply at openinfosecfoundation.org wrote:
> This is an automated email from the git hooks/post-receive script. It was
> generated because a ref change was pushed to the repository containing
> the project "OISF".
>
> The branch, master has been updated
> via a556338936ad3cd2b0379a6985fb62084368d99e (commit)
> via 56b96363b860c3d388a8866ddea043918e633626 (commit)
> via 63c9a3ab856dad25cb13f8d208df872f7253cf04 (commit)
> via b3e167932121f50fd6d56818acd4f0cd07d420de (commit)
> via 53df3982a1e3b15cb0506132dcf8bf5c9c077385 (commit)
> via 042fd850fc4238e9d16ed9c2458e9b4fa798eefc (commit)
> via f3fbc1a44c2c87df8da5874157834f503af55058 (commit)
> via 2ccd35c6e45de43d3bd02a3c718a4f6b3802dd9f (commit)
> via 33848124d193cec660f6c88e84b0d1be786c3fac (commit)
> via 96d20098b0c07ef55a4267930e5bcb54f924a04f (commit)
> via d59ca75e4697cb28d9a249c73a213770dbcf1702 (commit)
> via 04ea70ccf7c1074530bce57d0105d29d208b5f1e (commit)
> via 1c934acc850c53e9c4c035252e064139abc0745c (commit)
> via b402d97179eebe575158c6c60d4ae340b3b8aea6 (commit)
> via 66a3cd96a81f41fce1d0fc5104e6139362ceee2a (commit)
> via 417495e542a9d313300647a92084e27013b484dd (commit)
> via e1022ee5ae4f2583acf4ab8fec5936059a760379 (commit)
> via 27645f64c6db65b8e5872a9a3013d901074309d9 (commit)
> via 9b62ec65ab1b73051d573286e234d2383ec911f0 (commit)
> via 5945e652d6bcc539ef7288626bd5c04ed2e32a4c (commit)
> via f4a6f4b293ebd05c3088b95ceba18fb8978f128a (commit)
> via 23e01d23d314c41963d00fd04b30ef30721124b2 (commit)
> via 3e7baa6810755331253b1d69e5507adcadefcd28 (commit)
> via 403b2788d6357a05cc8e7e9cface4e7b7f3864fa (commit)
> via 59cda9a358b9092626469a343fcc4d5822537f7a (commit)
> via 64aee5e70c7a2720b307dc01ef72e545c867856f (commit)
> via ef0536794c99596e1f49f4ddbae73fe2e5241327 (commit)
> via 21acd72adf7ab6f5e544fdacf286fc7694a54eac (commit)
> via 70f0d3d2e79aad3d9142d6f76b438c8bd4744bf5 (commit)
> via 32fb9f375d1355d3dc902e972b31f8584cab6de2 (commit)
> via a6b7a560f149e3b63c248c69ac9dc75af0c4c0d3 (commit)
> via 7e3d537338757c7284b1f05ecb9efcd15ee23a54 (commit)
> via 1eef36b011aae14b8c88408ea1729f88a9cc3745 (commit)
> via 3c1edf3763f8fd571aa28578a481352765e5c6ec (commit)
> via cd618e48dfca132a8348c441d604f64d08eba24c (commit)
> via 4723f072543a5eac20278f7ae27a4d94c9dde07a (commit)
> via 9d5d46c4bb147e48d76be309864ce6d1f889da08 (commit)
> via a0ee6ade3ecd43a20f4d9ebb1331cfc77220b08a (commit)
> via 4537f889ef0f553df08ad8ab3dd45e16e31342ff (commit)
> via 222bc6e935361ef7f5eacbe9953dd4bcf24b4343 (commit)
> via 6d60b3a747940b6cc78be0dc5b0cd3b76b93ef09 (commit)
> via 06b1d71032f3b627058e4efe02dfd85ddb359094 (commit)
> from 0256ca242209edbea23948df52cef4db7fb0fc2e (commit)
>
> Those revisions listed above that are new to this repository have
> not appeared on any other notification email; so we list those
> revisions in full, below.
>
> - Log -----------------------------------------------------------------
> commit a556338936ad3cd2b0379a6985fb62084368d99e
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:40:09 2011 +0100
>
> Add magic-file example to suricta.yaml.
>
> commit 56b96363b860c3d388a8866ddea043918e633626
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:38:21 2011 +0100
>
> Fix merge artefact.
>
> commit 63c9a3ab856dad25cb13f8d208df872f7253cf04
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:36:26 2011 +0100
>
> Remove duplicate include.
>
> commit b3e167932121f50fd6d56818acd4f0cd07d420de
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:22:49 2011 +0100
>
> file handling: add example files.rules file
>
> Adding a rule file with various examples for using the fileext, filename,
> filemagic and filestore keywords.
>
> commit 53df3982a1e3b15cb0506132dcf8bf5c9c077385
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:18:39 2011 +0100
>
> Update suricata.yaml for file extraction.
>
> commit 042fd850fc4238e9d16ed9c2458e9b4fa798eefc
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:16:44 2011 +0100
>
> Make sure we check the sgh for no magic and no store once per flow direction.
>
> commit f3fbc1a44c2c87df8da5874157834f503af55058
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:07:08 2011 +0100
>
> file handling: filemagic matching improvement
>
> Magic buffer is a null terminated string. Allow matching on the final
> \0 using filemagic:"somevalue|00|"; so we can anchor to the end of the
> buffer.
>
> commit 2ccd35c6e45de43d3bd02a3c718a4f6b3802dd9f
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 29 15:06:49 2011 +0100
>
> Fix code after rebase.
>
> commit 33848124d193cec660f6c88e84b0d1be786c3fac
> Author: Victor Julien <victor at inliniac.net>
> Date: Mon Nov 28 20:15:02 2011 +0100
>
> Fix a multipart body parsing issue.
>
> commit 96d20098b0c07ef55a4267930e5bcb54f924a04f
> Author: Victor Julien <victor at inliniac.net>
> Date: Mon Nov 28 18:14:09 2011 +0100
>
> file inspect: stateful inspection split
>
> Split stateful detection of the files in a HTTP state between toserver
> and toclient inspection.
>
> commit d59ca75e4697cb28d9a249c73a213770dbcf1702
> Author: Victor Julien <victor at inliniac.net>
> Date: Mon Nov 28 17:44:55 2011 +0100
>
> file extract: split toserver and toclient tracking
>
> Split toserver and toclient file tracking for the http state.
>
> commit 04ea70ccf7c1074530bce57d0105d29d208b5f1e
> Author: Victor Julien <victor at inliniac.net>
> Date: Mon Nov 28 16:54:25 2011 +0100
>
> file extract: pruning
>
> Add pruning of files in memory so we keep only memory what we really need.
> Fix magic logic.
> Reset file part of the de_state on receiving another file in the same tx.
>
> commit 1c934acc850c53e9c4c035252e064139abc0745c
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Nov 15 09:54:35 2011 +0100
>
> Don't store fd per file (too many fd's). Enable IPv6 storing. Close file on receiving stream end flag.
>
> commit b402d97179eebe575158c6c60d4ae340b3b8aea6
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Nov 11 21:35:52 2011 +0100
>
> File carving -- enable reponse file extraction
>
> - Enable response body tracking
> - Enable file extraction for responses
> - File store meta file includes magic, close reason.
> - Option to force magic lookup for all stored files.
> - Fix libmagic calls thead safety.
>
> commit 66a3cd96a81f41fce1d0fc5104e6139362ceee2a
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Sep 27 23:28:35 2011 +0200
>
> Prepare HTTP response body tracking.
>
> commit 417495e542a9d313300647a92084e27013b484dd
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Sep 27 22:45:29 2011 +0200
>
> file-extraction: remove no longer used files.
>
> commit e1022ee5ae4f2583acf4ab8fec5936059a760379
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Sep 27 22:44:51 2011 +0200
>
> file-extraction: Disconnect file handling from flow and move into the app layer state.
>
> commit 27645f64c6db65b8e5872a9a3013d901074309d9
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Jul 1 13:14:42 2011 +0200
>
> Remove unused util-filetype.[ch] from Makefile.am.
>
> commit 9b62ec65ab1b73051d573286e234d2383ec911f0
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Jul 1 12:16:49 2011 +0200
>
> Make sure filemagic works properly regardless of filestore being in use for a flow.
>
> commit 5945e652d6bcc539ef7288626bd5c04ed2e32a4c
> Author: Victor Julien <victor at inliniac.net>
> Date: Wed Jun 29 16:57:30 2011 +0200
>
> Initial implementation of filemagic keyword.
>
> commit f4a6f4b293ebd05c3088b95ceba18fb8978f128a
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue Jun 28 15:19:30 2011 +0200
>
> Add libmagic detection, linking and a basic API.
>
> commit 23e01d23d314c41963d00fd04b30ef30721124b2
> Author: Victor Julien <victor at inliniac.net>
> Date: Wed Jun 22 18:38:14 2011 +0200
>
> Implement filestore keyword, including a way for the stateful detection engine to conclude that a file will never have to be stored.
>
> commit 3e7baa6810755331253b1d69e5507adcadefcd28
> Author: Victor Julien <victor at inliniac.net>
> Date: Wed Jun 8 16:43:52 2011 +0200
>
> Fix improper error handling in http body chunk function.
>
> commit 403b2788d6357a05cc8e7e9cface4e7b7f3864fa
> Author: Victor Julien <victor at inliniac.net>
> Date: Wed Jun 1 13:51:19 2011 +0200
>
> Add support for extracting PUT files.
>
> commit 59cda9a358b9092626469a343fcc4d5822537f7a
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue May 17 18:13:42 2011 +0200
>
> Fix not using new htp callback when using the bundled htp. Add indication to --build-info. Fix valgrind warning in test and further improve test.
>
> commit 64aee5e70c7a2720b307dc01ef72e545c867856f
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue May 3 22:46:32 2011 +0200
>
> Add file log to default suricata.yaml.
>
> commit ef0536794c99596e1f49f4ddbae73fe2e5241327
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue May 3 21:46:58 2011 +0200
>
> Adding comments, some cleanups.
>
> commit 21acd72adf7ab6f5e544fdacf286fc7694a54eac
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue May 3 17:55:34 2011 +0200
>
> Cleanups to the Multipart parsing code. Fixes to negation in filename and fileext.
>
> commit 70f0d3d2e79aad3d9142d6f76b438c8bd4744bf5
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue May 3 15:25:57 2011 +0200
>
> Add negation to filename and fileext, use same syntax as with content.
>
> commit 32fb9f375d1355d3dc902e972b31f8584cab6de2
> Author: Victor Julien <victor at inliniac.net>
> Date: Tue May 3 13:34:52 2011 +0200
>
> log-file log-dir option added, meta file created, fixes.
>
> commit a6b7a560f149e3b63c248c69ac9dc75af0c4c0d3
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Apr 29 10:48:53 2011 +0200
>
> Fix a bug in the HTTP file closing.
>
> commit 7e3d537338757c7284b1f05ecb9efcd15ee23a54
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Apr 29 09:10:45 2011 +0200
>
> Fix setting libhtp personality.
>
> commit 1eef36b011aae14b8c88408ea1729f88a9cc3745
> Author: Victor Julien <victor at inliniac.net>
> Date: Sun Apr 24 16:00:10 2011 +0200
>
> Initial checkin of a log-file module, that can write files extracted from flows to disk.
>
> commit 3c1edf3763f8fd571aa28578a481352765e5c6ec
> Author: Victor Julien <victor at inliniac.net>
> Date: Sun Apr 24 15:58:37 2011 +0200
>
> Add a file descriptor to the flow file structure.
>
> commit cd618e48dfca132a8348c441d604f64d08eba24c
> Author: Victor Julien <victor at inliniac.net>
> Date: Sun Apr 24 15:57:26 2011 +0200
>
> Allow for 0 (unlimited) HTTP request_body_limit, fix option parsing.
>
> commit 4723f072543a5eac20278f7ae27a4d94c9dde07a
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Apr 22 17:00:16 2011 +0200
>
> Improve testing and fix some bugs.
>
> commit 9d5d46c4bb147e48d76be309864ce6d1f889da08
> Author: Victor Julien <victor at inliniac.net>
> Date: Fri Apr 22 10:51:12 2011 +0200
>
> Implement flow file storage API, create HTP wrappers for it, use it in HTTP parsing.
>
> commit a0ee6ade3ecd43a20f4d9ebb1331cfc77220b08a
> Author: Victor Julien <victor at inliniac.net>
> Date: Thu Apr 21 12:50:25 2011 +0200
>
> Improve HTTP multipart parsing, add streaming parsing for files.
>
> commit 4537f889ef0f553df08ad8ab3dd45e16e31342ff
> Author: Victor Julien <victor at inliniac.net>
> Date: Sun Apr 17 17:18:09 2011 +0200
>
> Handle all strings as raw strings in HTTP content-type and content-disposition header parsing.
>
> commit 222bc6e935361ef7f5eacbe9953dd4bcf24b4343
> Author: System Administrator <root at macuto2.local>
> Date: Fri Apr 8 17:55:15 2011 +0200
>
> Flow files
>
> commit 6d60b3a747940b6cc78be0dc5b0cd3b76b93ef09
> Author: Pablo Rincon <pablo.rincon.crespo at gmail.com>
> Date: Wed Apr 6 17:23:52 2011 +0200
>
> filename and fileext keywords
>
> commit 06b1d71032f3b627058e4efe02dfd85ddb359094
> Author: Victor Julien <victor at inliniac.net>
> Date: Thu Apr 28 10:31:42 2011 +0200
>
> Small optimizations to IPV4 and TCP header parsing.
>
> -----------------------------------------------------------------------
>
> Summary of changes:
> configure.in | 29 +
> rules/files.rules | 45 +
> src/Makefile.am | 10 +
> src/app-layer-htp-body.c | 255 +++++
> src/{log-droplog.h => app-layer-htp-body.h} | 15 +-
> src/app-layer-htp-file.c | 790 ++++++++++++++
> src/{stream-tcp-inline.h => app-layer-htp-file.h} | 18 +-
> src/app-layer-htp.c | 1168 ++++++++++++++++++---
> src/app-layer-htp.h | 77 ++-
> src/app-layer-parser.c | 37 +
> src/app-layer-parser.h | 12 +-
> src/decode-tcp.c | 2 +-
> src/detect-engine-file.c | 233 ++++
> src/{flow-manager.h => detect-engine-file.h} | 11 +-
> src/detect-engine-hcbd.c | 16 +-
> src/detect-engine-siggroup.c | 53 +
> src/detect-engine-siggroup.h | 4 +
> src/detect-engine-state.c | 229 ++++-
> src/detect-engine-state.h | 78 +-
> src/detect-fileext.c | 295 ++++++
> src/{detect-ftpbounce.h => detect-fileext.h} | 17 +-
> src/detect-filemagic.c | 367 +++++++
> src/{stream-tcp-inline.h => detect-filemagic.h} | 22 +-
> src/detect-filename.c | 306 ++++++
> src/{detect-ftpbounce.h => detect-filename.h} | 18 +-
> src/detect-filestore.c | 131 +++
> src/{detect-metadata.h => detect-filestore.h} | 11 +-
> src/detect-http-client-body.c | 12 +-
> src/detect-parse.c | 195 ++++
> src/detect-parse.h | 2 +
> src/detect-pcre.c | 24 +-
> src/detect.c | 75 ++
> src/detect.h | 29 +-
> src/flow-util.h | 1 +
> src/flow.h | 8 +-
> src/log-file.c | 443 ++++++++
> src/{flow-manager.h => log-file.h} | 12 +-
> src/suricata.c | 14 +-
> src/tm-threads-common.h | 1 +
> src/util-error.h | 2 +
> src/util-magic.c | 533 ++++++++++
> src/{flow-manager.h => util-magic.h} | 14 +-
> src/util-spm-bm.h | 2 +-
> suricata.yaml | 31 +-
> 44 files changed, 5326 insertions(+), 321 deletions(-)
> create mode 100644 rules/files.rules
> create mode 100644 src/app-layer-htp-body.c
> copy src/{log-droplog.h => app-layer-htp-body.h} (64%)
> create mode 100644 src/app-layer-htp-file.c
> copy src/{stream-tcp-inline.h => app-layer-htp-file.h} (68%)
> create mode 100644 src/detect-engine-file.c
> copy src/{flow-manager.h => detect-engine-file.h} (74%)
> create mode 100644 src/detect-fileext.c
> copy src/{detect-ftpbounce.h => detect-fileext.h} (71%)
> create mode 100644 src/detect-filemagic.c
> copy src/{stream-tcp-inline.h => detect-filemagic.h} (65%)
> create mode 100644 src/detect-filename.c
> copy src/{detect-ftpbounce.h => detect-filename.h} (69%)
> create mode 100644 src/detect-filestore.c
> copy src/{detect-metadata.h => detect-filestore.h} (78%)
> create mode 100644 src/log-file.c
> copy src/{flow-manager.h => log-file.h} (77%)
> create mode 100644 src/util-magic.c
> copy src/{flow-manager.h => util-magic.h} (75%)
>
>
> hooks/post-receive
--
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------
More information about the Oisf-devel
mailing list