[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-3.1.2-103-g70c16f5
OISF Git
noreply at openinfosecfoundation.org
Thu Sep 22 16:12:44 UTC 2016
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".
The branch, master has been updated
via 70c16f50e733f6f7cc40c1bc3465eb966e3be517 (commit)
via aee1f0bb99b82bb2cfc6553ecae602ea81acaa05 (commit)
via 96427cf371da885fe2cec0106d77853025947a0d (commit)
via da8f3c987b0071555e5df79397985f87b52d7b87 (commit)
via 5e3b61cc65b688a31b2bb25cbf674b77f4e92085 (commit)
via 2b2984dae9d418d869952dde01642e56bf617f1e (commit)
via 499e27de14c2d960eae60c1fc7f050c4422ca712 (commit)
via 9d48720f9a2590ce2d51d37433358bfda21c8579 (commit)
via 98092f63b531bae3f585016a700c92cb0c56eac7 (commit)
via bc370606fc031743d42455b0be28547138ee2115 (commit)
via 723a14c0fe4b1cb33c970988acef174d4d8d0492 (commit)
via 2780fba1d120a69ecc7286441f79a6dc65d3c60e (commit)
via 7004987670f1e810fcb4558db7d581c106b7d533 (commit)
via 30c853a3040966f3541017e0451480879eeec3b4 (commit)
via 487cdda93d1836acc33323c3b57135c1844a8f41 (commit)
via afc796a0998e3e8b14d89cc2bd108d651fe1b818 (commit)
via 7ce196e3bfd74898353d598b9b968ffdd844cc6a (commit)
via 4cdcada397aa4fe2d3d3a37f1268c409205b2134 (commit)
via f7481c407820430669d89e2b7a36b96e08d796a2 (commit)
via 4426f3ff5522c60a32dc1e900b0a478ef70f5146 (commit)
via 2f5663dfe9bb9e8f6c85869b0aa1e76f0efede72 (commit)
via 53ebe4c5380781c740106e445e7782e3cbfee0a5 (commit)
via 89eb935f733c6b7d1521d140226ae1b2c251853a (commit)
via a6d928e269db561c223a31e5677f298708541009 (commit)
via 188b382c46a13f266843b9546c3b9daf03b9dc02 (commit)
from f1b550d9732ee41fc81288cc2e89ca1ebda3a096 (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
commit 70c16f50e733f6f7cc40c1bc3465eb966e3be517
Author: Victor Julien <victor at inliniac.net>
Date: Sat May 14 08:56:49 2016 +0200
flow-manager: optimize hash walking
Until now the flow manager would walk the entire flow hash table on an
interval. It would thus touch all flows, leading to a lot of memory
and cache pressure. In scenario's where the number of tracked flows run
into the hundreds on thousands, and the memory used can run into many
hundreds of megabytes or even gigabytes, this would lead to serious
performance degradation.
This patch introduces a new approach. A timestamp per flow bucket
(hash row) is maintained by the flow manager. It holds the timestamp
of the earliest possible timeout of a flow in the list. The hash walk
skips rows with timestamps beyond the current time.
As the timestamp depends on the flows in the hash row's list, and on
the 'state' of each flow in the list, any addition of a flow or
changing of a flow's state invalidates the timestamp. The flow manager
then has to walk the list again to set a new timestamp.
A utility function FlowUpdateState is introduced to change Flow states,
taking care of the bucket timestamp invalidation while at it.
Empty flow buckets use a special value so that we don't have to take
the flow bucket lock to find out the bucket is empty.
This patch also adds more performance counters:
flow_mgr.flows_checked | Total | 929
flow_mgr.flows_notimeout | Total | 391
flow_mgr.flows_timeout | Total | 538
flow_mgr.flows_removed | Total | 277
flow_mgr.flows_timeout_inuse | Total | 261
flow_mgr.rows_checked | Total | 1000000
flow_mgr.rows_skipped | Total | 998835
flow_mgr.rows_empty | Total | 290
flow_mgr.rows_maxlen | Total | 2
flow_mgr.flows_checked: number of flows checked for timeout in the
last pass
flow_mgr.flows_notimeout: number of flows out of flow_mgr.flows_checked
that didn't time out
flow_mgr.flows_timeout: number of out of flow_mgr.flows_checked that
did reach the time out
flow_mgr.flows_removed: number of flows out of flow_mgr.flows_timeout
that were really removed
flow_mgr.flows_timeout_inuse: number of flows out of flow_mgr.flows_timeout
that were still in use or needed work
flow_mgr.rows_checked: hash table rows checked
flow_mgr.rows_skipped: hash table rows skipped because non of the flows
would time out anyway
The counters below are only relating to rows that were not skipped.
flow_mgr.rows_empty: empty hash rows
flow_mgr.rows_maxlen: max number of flows per hash row. Best to keep low,
so increase hash-size if needed.
flow_mgr.rows_busy: row skipped because it was locked by another thread
commit aee1f0bb99b82bb2cfc6553ecae602ea81acaa05
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 13 23:04:30 2016 +0200
flow: simplify timeout logic
Instead of a single big FlowProto array containing timeouts separately
for normal and emergency cases, plus the 'Free' pointer for the
protoctx, split up these arrays.
An array made of FlowProtoTimeout for just the normal timeouts and an
mirror of that for emergency timeouts are used through a pointer that
will be set at init and by swapped by the emergency logic. It's swapped
back when the emergency is over.
The free funcs are moved to their own array.
This simplifies the timeout lookup code and shrinks the data that is
commonly used.
commit 96427cf371da885fe2cec0106d77853025947a0d
Author: Victor Julien <victor at inliniac.net>
Date: Fri May 13 22:13:59 2016 +0200
flow: remove dead code
commit da8f3c987b0071555e5df79397985f87b52d7b87
Author: Victor Julien <victor at inliniac.net>
Date: Thu Sep 22 12:38:54 2016 +0200
offloading: make disabling offloading configurable
Add a generic 'capture' section to the YAML:
# general settings affecting packet capture
capture:
# disable NIC offloading. It's restored when Suricata exists.
# Enabled by default
#disable-offloading: false
#
# disable checksum validation. Same as setting '-k none' on the
# commandline
#checksum-validation: none
commit 5e3b61cc65b688a31b2bb25cbf674b77f4e92085
Author: Victor Julien <victor at inliniac.net>
Date: Tue Jun 21 10:05:40 2016 +0200
offloading: reduce verbosity to 'perf'
commit 2b2984dae9d418d869952dde01642e56bf617f1e
Author: Victor Julien <victor at inliniac.net>
Date: Tue Jun 21 07:59:51 2016 +0200
offloading: implement restoring settings for BSD
commit 499e27de14c2d960eae60c1fc7f050c4422ca712
Author: Victor Julien <victor at inliniac.net>
Date: Tue Jun 21 09:35:33 2016 +0200
offloading: restore settings on exit
commit 9d48720f9a2590ce2d51d37433358bfda21c8579
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jun 20 23:02:02 2016 +0200
af-packet: optionally disable offloading
commit 98092f63b531bae3f585016a700c92cb0c56eac7
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jun 20 23:00:38 2016 +0200
offloading: Linux ethtool offloading support
commit bc370606fc031743d42455b0be28547138ee2115
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jun 20 20:18:43 2016 +0200
pcap: optionally disable offloading
commit 723a14c0fe4b1cb33c970988acef174d4d8d0492
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jun 20 20:17:30 2016 +0200
netmap: optionally disable offloading
commit 2780fba1d120a69ecc7286441f79a6dc65d3c60e
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jun 20 20:15:37 2016 +0200
device: add global flag for disabling offloading
Add global flag to disable offloading or just warn on it.
commit 7004987670f1e810fcb4558db7d581c106b7d533
Author: Victor Julien <victor at inliniac.net>
Date: Mon Jun 20 20:11:55 2016 +0200
offloading: preparation for disabling offload on BSD
Add functions for setting IFCAP flags.
commit 30c853a3040966f3541017e0451480879eeec3b4
Author: Jason Ish <ish at unx.ca>
Date: Wed Sep 21 14:19:55 2016 -0600
detect-ssl-state: use new unit test macros
commit 487cdda93d1836acc33323c3b57135c1844a8f41
Author: Jason Ish <jason.ish at emulex.com>
Date: Tue Sep 30 16:20:56 2014 -0600
ssl: issue 1231 - support ssl state negation
Snort compatible SSL state negation. Adds "," as a state
separator, but keeps "|" for compatibility with existing
Suricata rules.
commit afc796a0998e3e8b14d89cc2bd108d651fe1b818
Author: Jason Ish <jason.ish at emulex.com>
Date: Wed Oct 1 23:27:39 2014 -0600
ssl: store current state separately from cumulative state
The ssl_state keyword needs the current state, not the cumulative state
in order be compatible with Snort's implementation.
commit 7ce196e3bfd74898353d598b9b968ffdd844cc6a
Author: Jason Ish <ish at unx.ca>
Date: Wed Sep 21 09:43:42 2016 -0600
detect-pcre: use new unit test macros
commit 4cdcada397aa4fe2d3d3a37f1268c409205b2134
Author: Jason Ish <ish at unx.ca>
Date: Mon Sep 19 10:45:05 2016 -0600
pcre: fix missing quote in pcre unit test
commit f7481c407820430669d89e2b7a36b96e08d796a2
Author: Victor Julien <victor at inliniac.net>
Date: Thu Sep 22 10:45:29 2016 +0200
file-hashing: restore 'force-md5'
We don't want to break existing setups.
Do issue a warning that a new option is available.
commit 4426f3ff5522c60a32dc1e900b0a478ef70f5146
Author: Victor Julien <victor at inliniac.net>
Date: Thu Sep 22 10:26:56 2016 +0200
file: introduce common flags handling function
commit 2f5663dfe9bb9e8f6c85869b0aa1e76f0efede72
Author: Victor Julien <victor at inliniac.net>
Date: Thu Sep 22 10:26:12 2016 +0200
common: introduce BIT_U16
commit 53ebe4c5380781c740106e445e7782e3cbfee0a5
Author: Duarte Silva <development at serializing.me>
Date: Tue May 24 19:58:13 2016 +0200
file-hashing: added configuration options and common parsing code
commit 89eb935f733c6b7d1521d140226ae1b2c251853a
Author: Duarte Silva <development at serializing.me>
Date: Fri Apr 29 22:23:55 2016 +0200
file-hashing: added support for SHA-256 file hashing
commit a6d928e269db561c223a31e5677f298708541009
Author: Duarte Silva <development at serializing.me>
Date: Fri Apr 29 21:51:12 2016 +0200
file-hashing: added support for SHA-1 file hashing
commit 188b382c46a13f266843b9546c3b9daf03b9dc02
Author: Duarte Silva <development at serializing.me>
Date: Fri Apr 29 21:23:12 2016 +0200
file-hashing: common code added
Moved and adapted code from detect-filemd5 to util-detect-file-hash,
generalised code to work with SHA-1 and SHA-256 and added necessary
flags and other constants.
-----------------------------------------------------------------------
Summary of changes:
src/Makefile.am | 3 +
src/app-layer-htp-file.c | 34 +-
src/app-layer-smtp.c | 18 +-
src/app-layer-ssl.c | 40 +-
src/app-layer-ssl.h | 2 +
src/detect-engine-file.c | 12 +
src/detect-engine-siggroup.c | 18 +-
src/detect-engine-siggroup.h | 2 +-
src/detect-filemd5.c | 284 +----
src/detect-filemd5.h | 7 -
src/detect-filesha1.c | 161 +++
src/{detect-distance.h => detect-filesha1.h} | 12 +-
src/detect-filesha256.c | 161 +++
src/{detect-distance.h => detect-filesha256.h} | 12 +-
src/detect-pcre.c | 1378 +++++----------------
src/detect-ssl-state.c | 353 +++---
src/detect-ssl-state.h | 1 +
src/detect.c | 73 +-
src/detect.h | 28 +-
src/flow-hash.c | 7 +
src/flow-hash.h | 6 +
src/flow-manager.c | 180 ++-
src/flow-manager.h | 4 +
src/flow-private.h | 7 +-
src/flow.c | 255 ++--
src/flow.h | 56 +-
src/log-file.c | 27 +-
src/log-filestore.c | 26 +-
src/output-json-file.c | 28 +-
src/runmode-af-packet.c | 11 +-
src/runmode-netmap.c | 7 +
src/source-netmap.c | 3 -
src/source-pcap.c | 6 +
src/stream-tcp.c | 4 +-
src/suricata-common.h | 1 +
src/suricata.c | 21 +-
src/util-detect-file-hash.c | 368 ++++++
src/{detect-filemd5.h => util-detect-file-hash.h} | 22 +-
src/util-device.c | 23 +-
src/util-device.h | 16 +-
src/util-error.c | 3 +
src/util-error.h | 3 +
src/util-file.c | 286 ++++-
src/util-file.h | 42 +-
src/util-ioctl.c | 292 +++++
src/util-ioctl.h | 4 +
src/util-lua-common.c | 24 +
suricata.yaml.in | 22 +-
48 files changed, 2434 insertions(+), 1919 deletions(-)
create mode 100644 src/detect-filesha1.c
copy src/{detect-distance.h => detect-filesha1.h} (74%)
create mode 100644 src/detect-filesha256.c
copy src/{detect-distance.h => detect-filesha256.h} (74%)
create mode 100644 src/util-detect-file-hash.c
copy src/{detect-filemd5.h => util-detect-file-hash.h} (55%)
hooks/post-receive
--
OISF
More information about the Oisf-devel
mailing list