[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-3.1.2-103-g70c16f5

OISF Git noreply at openinfosecfoundation.org
Thu Sep 22 16:12:44 UTC 2016

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".

The branch, master has been updated
       via  70c16f50e733f6f7cc40c1bc3465eb966e3be517 (commit)
       via  aee1f0bb99b82bb2cfc6553ecae602ea81acaa05 (commit)
       via  96427cf371da885fe2cec0106d77853025947a0d (commit)
       via  da8f3c987b0071555e5df79397985f87b52d7b87 (commit)
       via  5e3b61cc65b688a31b2bb25cbf674b77f4e92085 (commit)
       via  2b2984dae9d418d869952dde01642e56bf617f1e (commit)
       via  499e27de14c2d960eae60c1fc7f050c4422ca712 (commit)
       via  9d48720f9a2590ce2d51d37433358bfda21c8579 (commit)
       via  98092f63b531bae3f585016a700c92cb0c56eac7 (commit)
       via  bc370606fc031743d42455b0be28547138ee2115 (commit)
       via  723a14c0fe4b1cb33c970988acef174d4d8d0492 (commit)
       via  2780fba1d120a69ecc7286441f79a6dc65d3c60e (commit)
       via  7004987670f1e810fcb4558db7d581c106b7d533 (commit)
       via  30c853a3040966f3541017e0451480879eeec3b4 (commit)
       via  487cdda93d1836acc33323c3b57135c1844a8f41 (commit)
       via  afc796a0998e3e8b14d89cc2bd108d651fe1b818 (commit)
       via  7ce196e3bfd74898353d598b9b968ffdd844cc6a (commit)
       via  4cdcada397aa4fe2d3d3a37f1268c409205b2134 (commit)
       via  f7481c407820430669d89e2b7a36b96e08d796a2 (commit)
       via  4426f3ff5522c60a32dc1e900b0a478ef70f5146 (commit)
       via  2f5663dfe9bb9e8f6c85869b0aa1e76f0efede72 (commit)
       via  53ebe4c5380781c740106e445e7782e3cbfee0a5 (commit)
       via  89eb935f733c6b7d1521d140226ae1b2c251853a (commit)
       via  a6d928e269db561c223a31e5677f298708541009 (commit)
       via  188b382c46a13f266843b9546c3b9daf03b9dc02 (commit)
      from  f1b550d9732ee41fc81288cc2e89ca1ebda3a096 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit 70c16f50e733f6f7cc40c1bc3465eb966e3be517
Author: Victor Julien <victor at inliniac.net>
Date:   Sat May 14 08:56:49 2016 +0200

    flow-manager: optimize hash walking
    Until now the flow manager would walk the entire flow hash table on an
    interval. It would thus touch all flows, leading to a lot of memory
    and cache pressure. In scenario's where the number of tracked flows run
    into the hundreds on thousands, and the memory used can run into many
    hundreds of megabytes or even gigabytes, this would lead to serious
    performance degradation.
    This patch introduces a new approach. A timestamp per flow bucket
    (hash row) is maintained by the flow manager. It holds the timestamp
    of the earliest possible timeout of a flow in the list. The hash walk
    skips rows with timestamps beyond the current time.
    As the timestamp depends on the flows in the hash row's list, and on
    the 'state' of each flow in the list, any addition of a flow or
    changing of a flow's state invalidates the timestamp. The flow manager
    then has to walk the list again to set a new timestamp.
    A utility function FlowUpdateState is introduced to change Flow states,
    taking care of the bucket timestamp invalidation while at it.
    Empty flow buckets use a special value so that we don't have to take
    the flow bucket lock to find out the bucket is empty.
    This patch also adds more performance counters:
    flow_mgr.flows_checked         | Total    | 929
    flow_mgr.flows_notimeout       | Total    | 391
    flow_mgr.flows_timeout         | Total    | 538
    flow_mgr.flows_removed         | Total    | 277
    flow_mgr.flows_timeout_inuse   | Total    | 261
    flow_mgr.rows_checked          | Total    | 1000000
    flow_mgr.rows_skipped          | Total    | 998835
    flow_mgr.rows_empty            | Total    | 290
    flow_mgr.rows_maxlen           | Total    | 2
    flow_mgr.flows_checked: number of flows checked for timeout in the
                            last pass
    flow_mgr.flows_notimeout: number of flows out of flow_mgr.flows_checked
                            that didn't time out
    flow_mgr.flows_timeout: number of out of flow_mgr.flows_checked that
                            did reach the time out
    flow_mgr.flows_removed: number of flows out of flow_mgr.flows_timeout
                            that were really removed
    flow_mgr.flows_timeout_inuse: number of flows out of flow_mgr.flows_timeout
                            that were still in use or needed work
    flow_mgr.rows_checked: hash table rows checked
    flow_mgr.rows_skipped: hash table rows skipped because non of the flows
                            would time out anyway
    The counters below are only relating to rows that were not skipped.
    flow_mgr.rows_empty:   empty hash rows
    flow_mgr.rows_maxlen:  max number of flows per hash row. Best to keep low,
                            so increase hash-size if needed.
    flow_mgr.rows_busy:    row skipped because it was locked by another thread

commit aee1f0bb99b82bb2cfc6553ecae602ea81acaa05
Author: Victor Julien <victor at inliniac.net>
Date:   Fri May 13 23:04:30 2016 +0200

    flow: simplify timeout logic
    Instead of a single big FlowProto array containing timeouts separately
    for normal and emergency cases, plus the 'Free' pointer for the
    protoctx, split up these arrays.
    An array made of FlowProtoTimeout for just the normal timeouts and an
    mirror of that for emergency timeouts are used through a pointer that
    will be set at init and by swapped by the emergency logic. It's swapped
    back when the emergency is over.
    The free funcs are moved to their own array.
    This simplifies the timeout lookup code and shrinks the data that is
    commonly used.

commit 96427cf371da885fe2cec0106d77853025947a0d
Author: Victor Julien <victor at inliniac.net>
Date:   Fri May 13 22:13:59 2016 +0200

    flow: remove dead code

commit da8f3c987b0071555e5df79397985f87b52d7b87
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Sep 22 12:38:54 2016 +0200

    offloading: make disabling offloading configurable
    Add a generic 'capture' section to the YAML:
      # general settings affecting packet capture
        # disable NIC offloading. It's restored when Suricata exists.
        # Enabled by default
        #disable-offloading: false
        # disable checksum validation. Same as setting '-k none' on the
        # commandline
        #checksum-validation: none

commit 5e3b61cc65b688a31b2bb25cbf674b77f4e92085
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Jun 21 10:05:40 2016 +0200

    offloading: reduce verbosity to 'perf'

commit 2b2984dae9d418d869952dde01642e56bf617f1e
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Jun 21 07:59:51 2016 +0200

    offloading: implement restoring settings for BSD

commit 499e27de14c2d960eae60c1fc7f050c4422ca712
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Jun 21 09:35:33 2016 +0200

    offloading: restore settings on exit

commit 9d48720f9a2590ce2d51d37433358bfda21c8579
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Jun 20 23:02:02 2016 +0200

    af-packet: optionally disable offloading

commit 98092f63b531bae3f585016a700c92cb0c56eac7
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Jun 20 23:00:38 2016 +0200

    offloading: Linux ethtool offloading support

commit bc370606fc031743d42455b0be28547138ee2115
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Jun 20 20:18:43 2016 +0200

    pcap: optionally disable offloading

commit 723a14c0fe4b1cb33c970988acef174d4d8d0492
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Jun 20 20:17:30 2016 +0200

    netmap: optionally disable offloading

commit 2780fba1d120a69ecc7286441f79a6dc65d3c60e
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Jun 20 20:15:37 2016 +0200

    device: add global flag for disabling offloading
    Add global flag to disable offloading or just warn on it.

commit 7004987670f1e810fcb4558db7d581c106b7d533
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Jun 20 20:11:55 2016 +0200

    offloading: preparation for disabling offload on BSD
    Add functions for setting IFCAP flags.

commit 30c853a3040966f3541017e0451480879eeec3b4
Author: Jason Ish <ish at unx.ca>
Date:   Wed Sep 21 14:19:55 2016 -0600

    detect-ssl-state: use new unit test macros

commit 487cdda93d1836acc33323c3b57135c1844a8f41
Author: Jason Ish <jason.ish at emulex.com>
Date:   Tue Sep 30 16:20:56 2014 -0600

    ssl: issue 1231 - support ssl state negation
    Snort compatible SSL state negation. Adds "," as a state
    separator, but keeps "|" for compatibility with existing
    Suricata rules.

commit afc796a0998e3e8b14d89cc2bd108d651fe1b818
Author: Jason Ish <jason.ish at emulex.com>
Date:   Wed Oct 1 23:27:39 2014 -0600

    ssl: store current state separately from cumulative state
    The ssl_state keyword needs the current state, not the cumulative state
    in order be compatible with Snort's implementation.

commit 7ce196e3bfd74898353d598b9b968ffdd844cc6a
Author: Jason Ish <ish at unx.ca>
Date:   Wed Sep 21 09:43:42 2016 -0600

    detect-pcre: use new unit test macros

commit 4cdcada397aa4fe2d3d3a37f1268c409205b2134
Author: Jason Ish <ish at unx.ca>
Date:   Mon Sep 19 10:45:05 2016 -0600

    pcre: fix missing quote in pcre unit test

commit f7481c407820430669d89e2b7a36b96e08d796a2
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Sep 22 10:45:29 2016 +0200

    file-hashing: restore 'force-md5'
    We don't want to break existing setups.
    Do issue a warning that a new option is available.

commit 4426f3ff5522c60a32dc1e900b0a478ef70f5146
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Sep 22 10:26:56 2016 +0200

    file: introduce common flags handling function

commit 2f5663dfe9bb9e8f6c85869b0aa1e76f0efede72
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Sep 22 10:26:12 2016 +0200

    common: introduce BIT_U16

commit 53ebe4c5380781c740106e445e7782e3cbfee0a5
Author: Duarte Silva <development at serializing.me>
Date:   Tue May 24 19:58:13 2016 +0200

    file-hashing: added configuration options and common parsing code

commit 89eb935f733c6b7d1521d140226ae1b2c251853a
Author: Duarte Silva <development at serializing.me>
Date:   Fri Apr 29 22:23:55 2016 +0200

    file-hashing: added support for SHA-256 file hashing

commit a6d928e269db561c223a31e5677f298708541009
Author: Duarte Silva <development at serializing.me>
Date:   Fri Apr 29 21:51:12 2016 +0200

    file-hashing: added support for SHA-1 file hashing

commit 188b382c46a13f266843b9546c3b9daf03b9dc02
Author: Duarte Silva <development at serializing.me>
Date:   Fri Apr 29 21:23:12 2016 +0200

    file-hashing: common code added
    Moved and adapted code from detect-filemd5 to util-detect-file-hash,
    generalised code to work with SHA-1 and SHA-256 and added necessary
    flags and other constants.


Summary of changes:
 src/Makefile.am                                   |    3 +
 src/app-layer-htp-file.c                          |   34 +-
 src/app-layer-smtp.c                              |   18 +-
 src/app-layer-ssl.c                               |   40 +-
 src/app-layer-ssl.h                               |    2 +
 src/detect-engine-file.c                          |   12 +
 src/detect-engine-siggroup.c                      |   18 +-
 src/detect-engine-siggroup.h                      |    2 +-
 src/detect-filemd5.c                              |  284 +----
 src/detect-filemd5.h                              |    7 -
 src/detect-filesha1.c                             |  161 +++
 src/{detect-distance.h => detect-filesha1.h}      |   12 +-
 src/detect-filesha256.c                           |  161 +++
 src/{detect-distance.h => detect-filesha256.h}    |   12 +-
 src/detect-pcre.c                                 | 1378 +++++----------------
 src/detect-ssl-state.c                            |  353 +++---
 src/detect-ssl-state.h                            |    1 +
 src/detect.c                                      |   73 +-
 src/detect.h                                      |   28 +-
 src/flow-hash.c                                   |    7 +
 src/flow-hash.h                                   |    6 +
 src/flow-manager.c                                |  180 ++-
 src/flow-manager.h                                |    4 +
 src/flow-private.h                                |    7 +-
 src/flow.c                                        |  255 ++--
 src/flow.h                                        |   56 +-
 src/log-file.c                                    |   27 +-
 src/log-filestore.c                               |   26 +-
 src/output-json-file.c                            |   28 +-
 src/runmode-af-packet.c                           |   11 +-
 src/runmode-netmap.c                              |    7 +
 src/source-netmap.c                               |    3 -
 src/source-pcap.c                                 |    6 +
 src/stream-tcp.c                                  |    4 +-
 src/suricata-common.h                             |    1 +
 src/suricata.c                                    |   21 +-
 src/util-detect-file-hash.c                       |  368 ++++++
 src/{detect-filemd5.h => util-detect-file-hash.h} |   22 +-
 src/util-device.c                                 |   23 +-
 src/util-device.h                                 |   16 +-
 src/util-error.c                                  |    3 +
 src/util-error.h                                  |    3 +
 src/util-file.c                                   |  286 ++++-
 src/util-file.h                                   |   42 +-
 src/util-ioctl.c                                  |  292 +++++
 src/util-ioctl.h                                  |    4 +
 src/util-lua-common.c                             |   24 +
 suricata.yaml.in                                  |   22 +-
 48 files changed, 2434 insertions(+), 1919 deletions(-)
 create mode 100644 src/detect-filesha1.c
 copy src/{detect-distance.h => detect-filesha1.h} (74%)
 create mode 100644 src/detect-filesha256.c
 copy src/{detect-distance.h => detect-filesha256.h} (74%)
 create mode 100644 src/util-detect-file-hash.c
 copy src/{detect-filemd5.h => util-detect-file-hash.h} (55%)


More information about the Oisf-devel mailing list