[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-4.0.1-268-gd05355d
OISF Git
noreply at openinfosecfoundation.org
Thu Jan 18 21:00:57 UTC 2018
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".
The branch, master has been updated
via d05355db3d6e2752ae0582a7ea8c1a0f08bde91c (commit)
via 46d754044ed39e3f90543bedd7b64e966d44c521 (commit)
via 5420c0ab062627782591c1801089536edffe644f (commit)
via aa0760a8d5527510a16d1a30c4fae609870da6f2 (commit)
via cc35a5b81ffde8cdb15d2ada7aea96587a853462 (commit)
via 9b1d26807151075de14f45a41c010dd183cfa662 (commit)
via 9456a3164db2a165a864ca2bd09edd7930c5a105 (commit)
via 50b5a3a56d3b4623d2cf193c2e796fb345385dac (commit)
via f7c3f3018664838d8c70c643c74a1a98f1a83972 (commit)
via f631e8cd906d26a9c9ecdc3f5c9e0c3f8d1f05dd (commit)
via 4a9731505717cab8a0fbc68f6a649dfb005c0120 (commit)
via dbdac73784c7420488ea2e83bb5bf2c8c60d5c20 (commit)
via ccbe7401b6ad171cc95c37486f826a2aa2d877b5 (commit)
via 59bb98afccee9f8af8f73841bc4e2cf829f5b5cd (commit)
via c8b6212a975d05a57edee70c8eb5cbb577e9e21a (commit)
via 38bbdb51d5c102a8f376df943d2f6b5f5de81ab6 (commit)
via c0ffe4055a579e895ab50b9e06db4ef18d84bb6e (commit)
via 0d558ddc27a4f74b8754b7bd0df7bc9f67c8b94c (commit)
via de2fffca5e5d3aece0176418fff3a253eaab4aed (commit)
via 00e6cd4ced0197c580ebea8855c13889409bab71 (commit)
from 50a762bfd164e27c55624c7e4f7e3c2e064ca788 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit d05355db3d6e2752ae0582a7ea8c1a0f08bde91c
Author: Victor Julien <victor at inliniac.net>
Date: Thu Jan 18 17:38:09 2018 +0100
filestore: minor cleanups and warning fixes
commit 46d754044ed39e3f90543bedd7b64e966d44c521
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 18 06:42:19 2018 -0600
suricatasc: don't use find -delete
For when -delete isn't supported by find. Instead use
-print0 with xargs -0.
commit 5420c0ab062627782591c1801089536edffe644f
Author: Jason Ish <ish at unx.ca>
Date: Wed Jan 17 07:51:38 2018 -0600
doc: document file-store v2
commit aa0760a8d5527510a16d1a30c4fae609870da6f2
Author: Jason Ish <ish at unx.ca>
Date: Fri Jan 12 14:46:31 2018 -0600
filestore: only allow one filestore to be enabled
There is probably not too much bad about enabling both, but
open file counts can get messy with both enabled. And v1
should be schedule for deprecation soon enough.
commit cc35a5b81ffde8cdb15d2ada7aea96587a853462
Author: Jason Ish <ish at unx.ca>
Date: Fri Jan 12 14:43:01 2018 -0600
filestore (old): register global stat in init func
This doesn't need to be registered from suricata.c. And moving
it to the init function makes sure its only registered if
the logger is actually enabled.
commit 9b1d26807151075de14f45a41c010dd183cfa662
Author: Jason Ish <ish at unx.ca>
Date: Fri Jan 12 13:35:51 2018 -0600
filestore2: warn once for file errors
Track each type of error warning and only log it once. Also create
a new stat, file_store.fs_errors to count each file system type
error (open, rename, unlink).
Also remove exit stats, they are of limited value.
commit 9456a3164db2a165a864ca2bd09edd7930c5a105
Author: Jason Ish <ish at unx.ca>
Date: Fri Jan 12 12:56:42 2018 -0600
util-error: define SC_ERR_MAX
commit 50b5a3a56d3b4623d2cf193c2e796fb345385dac
Author: Jason Ish <ish at unx.ca>
Date: Tue Jan 9 07:51:26 2018 -0600
suricatactl: a new python script for misc. tasks
Use a new directory, Python to host the Suricata python modules.
One entry point is suricatactl, a control script for
miscalleneous tasks. Currently onl filestore pruning
is implemented.
commit f7c3f3018664838d8c70c643c74a1a98f1a83972
Author: Jason Ish <ish at unx.ca>
Date: Mon Jan 8 14:09:01 2018 -0600
filestore v2: use fileinfo records as metadata
As fileinfo records are logged to the main eve log, disable
metadata by default. But when enabled, just use the fileinfo
record.
Metadata is stored in a file named:
<sha256>.<seconds>.<file_id>.json
where the sha256 is the same as the file logged, the seconds
is the unix timestamp in seconds for the fileinfo record,
and the file_id is an atomically incremented integer per
Suricata instance.
This should allow for each occurrence of the same file to have
its own metadata file. But a collision is expected when running
Suricata repeatedly over the same pcap, as that would be the
exact same occurrence of a file.
commit f631e8cd906d26a9c9ecdc3f5c9e0c3f8d1f05dd
Author: Jason Ish <ish at unx.ca>
Date: Mon Jan 8 13:39:56 2018 -0600
file extract: force sha256 even if truncated
Even if a file is truncated, force the SHA256 if force sha256
is set to yes.
The new file store requires the sha256 regardless of the file
state if it is to be logged, as the filename is based on the
sha256.
commit 4a9731505717cab8a0fbc68f6a649dfb005c0120
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 4 14:28:29 2018 -0600
filestore v2 - initial version
Filestore v2 is starts as a copy of log-filestore with the
following changes.
- NSS is required as file names as based on the SHA256.
- Work/tmp files are stored in a temp. directory, then
moved into a directory tree where the directory names
are the first 2 characters of the hex SHA256.
- Removes the need for a waldo file or pid in the filenames.
commit dbdac73784c7420488ea2e83bb5bf2c8c60d5c20
Author: Jason Ish <ish at unx.ca>
Date: Mon Jan 8 15:11:54 2018 -0600
configure: check for utime.h and utime()
commit ccbe7401b6ad171cc95c37486f826a2aa2d877b5
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 18 06:17:00 2018 -0600
output-json-file: let caller decide if file is stored
Mainly for the filestore module, which may have its own
knowledge of the file being stored before others.
commit 59bb98afccee9f8af8f73841bc4e2cf829f5b5cd
Author: Jason Ish <ish at unx.ca>
Date: Mon Jan 8 13:41:34 2018 -0600
eve/fileinfo: split record creation from writing
Split the building of the fileinfo record from the writing
of the record so the building can be called from other code.
Specifically the new filestore output which uses fileinfo
records as the metadata.
commit c8b6212a975d05a57edee70c8eb5cbb577e9e21a
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 4 11:14:42 2018 -0600
util-error: new error: SC_ERR_CREATE_DIRECTORY
For logging directory create errors.
commit 38bbdb51d5c102a8f376df943d2f6b5f5de81ab6
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 4 11:11:06 2018 -0600
SCPathExists - function to see if a path exists
Returns true if path exists, otherwise false.
commit c0ffe4055a579e895ab50b9e06db4ef18d84bb6e
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 4 11:07:50 2018 -0600
create directory: final arg to control full path or prefix
Give SCCreateDirectoryTree a new argument, final. If true the
full path will be created as a directory. If false, the last
component will not be created as a directory (current
behaviour).
commit 0d558ddc27a4f74b8754b7bd0df7bc9f67c8b94c
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 4 11:04:52 2018 -0600
create directory: fix strlcpy usage
The final character was being cut off.
commit de2fffca5e5d3aece0176418fff3a253eaab4aed
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 4 10:06:31 2018 -0600
util: move SCCreateDirectoryTree to util-path
Renames SCLogCreateDirectoryTree to SCCreateDirectoryTree
and move into a util module for re-use.
Also moves SCMkDir from suricata-common.h to the more
appropriately names util-path.h.
I would have prefered to use util-file for file related options
but that is already used by file store utilities. util-path
is close enough for file related operations.
commit 00e6cd4ced0197c580ebea8855c13889409bab71
Author: Jason Ish <ish at unx.ca>
Date: Thu Jan 11 16:34:33 2018 -0600
output: introduce init return type
The new OutputInitResult is a struct return type that allows
logger init functions to return a NULL context without
raising error.
Instead of returning NULL to signal error, the "ok" field will
be set to false. If ok, but the ctx is NULL, then silently
move on to the next logger.
Use case: multiple versions of a specific logger, and one
implementation decides the configuration is not for that
implemenation. It can return NULL, ok.
-----------------------------------------------------------------------
Summary of changes:
Makefile.am | 2 +-
configure.ac | 5 +-
doc/userguide/configuration/suricata-yaml.rst | 48 ++
doc/userguide/file-extraction/file-extraction.rst | 74 ++-
python/.gitignore | 3 +
python/Makefile.am | 29 ++
python/bin/suricatactl | 40 ++
python/setup.py | 32 ++
config.rpath => python/suricata/__init__.py | 0
config.rpath => python/suricata/ctl/__init__.py | 0
python/suricata/ctl/filestore.py | 118 +++++
python/suricata/ctl/loghandler.py | 79 ++++
python/suricata/ctl/main.py | 50 ++
python/suricata/ctl/test_filestore.py | 18 +
scripts/suricatasc/Makefile.am | 2 +-
src/Makefile.am | 1 +
src/alert-debuglog.c | 9 +-
src/alert-fastlog.c | 13 +-
src/alert-fastlog.h | 2 +-
src/alert-prelude.c | 21 +-
src/alert-syslog.c | 11 +-
src/alert-unified2-alert.c | 74 +--
src/alert-unified2-alert.h | 2 +-
src/log-dnslog.c | 15 +-
src/log-droplog.c | 15 +-
src/log-file.c | 14 +-
src/log-filestore.c | 33 +-
src/log-httplog.c | 15 +-
src/log-httplog.h | 2 +-
src/log-pcap.c | 9 +-
src/log-stats.c | 17 +-
src/log-tcp-data.c | 15 +-
src/log-tcp-data.h | 2 +-
src/log-tlslog.c | 11 +-
src/log-tlsstore.c | 10 +-
src/output-filestore.c | 503 +++++++++++++++++++++
...app-layer-dns-udp-rust.h => output-filestore.h} | 11 +-
src/output-json-alert.c | 26 +-
src/output-json-dnp3.c | 11 +-
src/output-json-dns.c | 26 +-
src/output-json-drop.c | 32 +-
src/output-json-file.c | 76 ++--
src/output-json-file.h | 5 +
src/output-json-flow.c | 26 +-
src/output-json-http.c | 26 +-
src/output-json-netflow.c | 26 +-
src/output-json-nfs.c | 11 +-
src/output-json-smtp.c | 26 +-
src/output-json-ssh.c | 28 +-
src/output-json-stats.c | 29 +-
src/output-json-template.c | 11 +-
src/output-json-tls.c | 26 +-
src/output-json-vars.c | 26 +-
src/output-json.c | 22 +-
src/output-json.h | 3 +-
src/output-lua.c | 30 +-
src/output.c | 2 +
src/output.h | 9 +-
src/runmode-af-packet.c | 2 +-
src/runmode-netmap.c | 2 +-
src/runmode-pcap.c | 2 +-
src/runmodes.c | 41 +-
src/suricata-common.h | 10 +-
src/suricata.c | 2 -
src/util-error.c | 2 +
src/util-error.h | 2 +
src/util-file.c | 27 +-
src/util-logopenfile.c | 38 +-
src/util-path.c | 73 +++
src/util-path.h | 9 +
src/util-runmodes.c | 2 +-
suricata.yaml.in | 49 +-
72 files changed, 1657 insertions(+), 386 deletions(-)
create mode 100644 python/.gitignore
create mode 100644 python/Makefile.am
create mode 100755 python/bin/suricatactl
create mode 100644 python/setup.py
copy config.rpath => python/suricata/__init__.py (100%)
copy config.rpath => python/suricata/ctl/__init__.py (100%)
create mode 100644 python/suricata/ctl/filestore.py
create mode 100644 python/suricata/ctl/loghandler.py
create mode 100644 python/suricata/ctl/main.py
create mode 100644 python/suricata/ctl/test_filestore.py
create mode 100644 src/output-filestore.c
copy src/{app-layer-dns-udp-rust.h => output-filestore.h} (73%)
hooks/post-receive
--
OISF
More information about the Oisf-devel
mailing list