[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-2.1beta3-19-gcbe9342

OISF Git noreply at openinfosecfoundation.org
Wed Feb 4 18:17:21 UTC 2015

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".

The branch, master has been updated
       via  cbe934267efa0973b042853cd36a99c2b3c83941 (commit)
       via  c58b2b4b18375781d56384c5c2e123c6a2fa3e31 (commit)
       via  e58fd3cc6e29312e957f8b78b43a7115883b4eb8 (commit)
       via  fbe6dac1ae013d4d118cbe3da2fe9351f19dd9d4 (commit)
       via  5251ea9ff5a8b419cb3f642606a036210c6e0bd1 (commit)
       via  5587372ce15cf3ead98867e1daf3dd010529b4ad (commit)
       via  a0732d3db23538689af9432e012d26241996b99b (commit)
       via  9327b08ab16c10807b70a2fb200d5681ed6aea39 (commit)
       via  37b56dca5570313a0bee6f19d1955f8baf40b926 (commit)
       via  2bba5eb7044504044b9a0e913e95fa4f542e49e4 (commit)
       via  8c1bc7cfb6adddfbfcbad6ee087b5fd45383cd66 (commit)
       via  6ca9c8eb32c5a44d52ae4b3d6ad0d61848025739 (commit)
       via  ff2fecf59099f3d1808d7d89ee2c459bdf59ab63 (commit)
       via  97908bcd2d4157536ba03fd724ed0ee6351ab73f (commit)
       via  e1d134b0275a1fef073cee55b75b43107164a851 (commit)
       via  29d2483efb47108efb26728483e94ea502ac7f8c (commit)
       via  e4cb8715de69b12fbb1605a387b79ff221bf9928 (commit)
       via  ed791a562eddbcd14b31063c6a1519c2453c3a76 (commit)
       via  e494336453c16c00f1026da63685ffdc2b9f567f (commit)
      from  a5641bc7c225d447d21a65927b4077f095d7dca2 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit cbe934267efa0973b042853cd36a99c2b3c83941
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Jan 23 10:22:55 2015 +0100

    file: register filedata log before file log
    This way the file log can log the 'stored' info that the filedata
    log sets.

commit c58b2b4b18375781d56384c5c2e123c6a2fa3e31
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Jan 23 10:22:01 2015 +0100

    file: improve file pruning
    Check if file has been logged/stored before considering it 'done'.

commit e58fd3cc6e29312e957f8b78b43a7115883b4eb8
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Jan 23 10:20:37 2015 +0100

    runmodes: add funcs to check if file loggers enabled
    Add functions to check if file/filedata loggers are enabled.

commit fbe6dac1ae013d4d118cbe3da2fe9351f19dd9d4
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Jan 22 19:24:35 2015 +0100

    file: optimize file pruning
    FilePrune would clear the files, but not free them and remove them
    from the list. This lead to ever growing lists in some cases.
    Especially in HTTP sessions with many transactions, this could slow
    us down.

commit 5251ea9ff5a8b419cb3f642606a036210c6e0bd1
Author: Victor Julien <victor at inliniac.net>
Date:   Sat Jan 24 14:10:11 2015 +0100

    flow: lockless flow manager checks
    Until this point, the flow manager would check for timed out flows
    by walking the flow hash, locking first the hash row and then each
    individual flow to get it's state and timestamp. To not be too
    intrusive trylocks were used so that a busy flow wouldn't cause the
    flow manager to wait for a long time while holding the hash row lock.
    Building on the changes in handling of the flow state and lastts
    fields, this patch changes the flow managers behavior.
    It can now get a flows state atomically and the lastts can be safely
    read while holding just the flow hash row lock. This allows the flow
    manager to do the basic time out check much more cheaply:
    1. it doesn't have to wait for getting a lock
    2. it doesn't interupt the packet path
    As a consequence the trylock is now also gone. A flow that returns
    'true' on timeout is pretty much certainly not going to be busy so
    we can safely lock it unconditionally. This also means the flow
    manager now walks the entire row unconditionally and is guaranteed
    to inspect each flow in the row.
    To make sure the functions called before the flow lock don't
    accidentally change the flow (which would require a lock) the args
    to these flows are changed to const pointers.

commit 5587372ce15cf3ead98867e1daf3dd010529b4ad
Author: Victor Julien <victor at inliniac.net>
Date:   Sat Jan 24 13:46:25 2015 +0100

    flow: modify lastts update logic
    In the lastts timeval struct field in the flow the timestamp of the
    last packet to update is recorded. This allows for tracking the timeout
    of the flow. So far, this value was updated under the flow lock and also
    read under the flow lock.
    This patch moves the updating of this field to the FlowGetFlowFromHash
    function, where it updated at the point where both the Flow and the
    Flow Hash Row lock are held. This guarantees that the field is only
    updated when both locks are held.
    This makes reading the field safe when either lock is held, which is the
    purpose of this patch.
    The flow manager, while holding the flow hash row lock, can now safely
    read the lastts value. This allows it to do the flow timeout check
    without actually locking the flow.

commit a0732d3db23538689af9432e012d26241996b99b
Author: Victor Julien <victor at inliniac.net>
Date:   Sat Jan 24 13:18:51 2015 +0100

    flow: change flow state logic
    A flow has 3 states: NEW, ESTABLISHED and CLOSED.
    For all protocols except TCP, a flow is in state NEW as long as just one
    side of the conversation has been seen. When both sides have been
    observed the state is moved to ESTABLISHED.
    TCP has a different logic, controlled by the stream engine. Here the TCP
    state is leading.
    Until now, when parts of the engine needed to know the flow state, it
    would invoke a per protocol callback 'GetProtoState'. For TCP this would
    return the state based on the TcpSession.
    This patch changes this logic. It introduces an atomic variable in the
    flow 'flow_state'. It defaults to NEW and is set to ESTABLISHED for non-
    TCP protocols when we've seen both sides of the conversation.
    For TCP, the state is updated from the TCP engine directly.
    The goal is to allow for access to the state without holding the Flow's
    main mutex lock. This will later allow the Flow Manager(s) to evaluate
    the Flow w/o interupting it.

commit 9327b08ab16c10807b70a2fb200d5681ed6aea39
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Jan 22 13:43:31 2015 +0100

    tcp: add stream.reassembly.zero-copy-size option
    The option sets in bytes the value at which segment data is passed to
    the app layer API directly. Data sizes equal to and higher than the
    value set are passed on directly.
    Default is 128.

commit 37b56dca5570313a0bee6f19d1955f8baf40b926
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Jan 16 12:12:38 2015 +0100

    tcp: add debug stats about reassembly fast paths
    Only shown if --enable-debug is passed to configure.

commit 2bba5eb7044504044b9a0e913e95fa4f542e49e4
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Jan 16 11:53:29 2015 +0100

    tcp: zero copy fast path in app-layer reassembly
    Create 2 'fast paths' for app layer reassembly. Both are about reducing
    copying. In the cases described below, we pass the segment's data
    directly to the app layer API, instead of first copying it into a buffer
    than we then pass. This safes a copy.
    The first is for the case when we have just one single segment that was
    just ack'd. As we know that we won't use any other segment this round,
    we can just use the segment data.
    The second case is more aggressive. When the segment meets a certain
    size limit (currently hardcoded at 128 bytes), we pass it to the
    app-layer API directly. Thus invoking the app-layer somewhat more often
    to safe some copies.

commit 8c1bc7cfb6adddfbfcbad6ee087b5fd45383cd66
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 19:08:14 2014 +0100

    stream: move raw stream gap handling into util func

commit 6ca9c8eb32c5a44d52ae4b3d6ad0d61848025739
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 19:01:03 2014 +0100

    stream: move raw reassembly into util func

commit ff2fecf59099f3d1808d7d89ee2c459bdf59ab63
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 15:39:49 2014 +0100

    stream: remove StreamTcpReassembleInlineAppLayer
    Function is now unused.

commit 97908bcd2d4157536ba03fd724ed0ee6351ab73f
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 15:27:58 2014 +0100

    stream: unify inline and non-inline applayer assembly
    Unifiy inline and non-inline app layer stream reassembly to aid
    maintainability of the code.

commit e1d134b0275a1fef073cee55b75b43107164a851
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 14:36:05 2014 +0100

    stream: remove STREAM_SET_FLAGS
    Use the unified StreamGetAppLayerFlags instead.

commit 29d2483efb47108efb26728483e94ea502ac7f8c
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 14:26:35 2014 +0100

    stream: update inline tests
    Make sure inline tests set the stream_inline flag.

commit e4cb8715de69b12fbb1605a387b79ff221bf9928
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 14:25:34 2014 +0100

    stream: replace STREAM_SET_INLINE_FLAGS macro
    Replace it by a generic function StreamGetAppLayerFlags, that can
    be used both by inline and non-inline.

commit ed791a562eddbcd14b31063c6a1519c2453c3a76
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 13:48:00 2014 +0100

    stream: track data sent to app-layer

commit e494336453c16c00f1026da63685ffdc2b9f567f
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Dec 9 12:56:55 2014 +0100

    stream: move reassembly loop into util funcs
    Move IDS per segment reassembly and gap handling into utility functions.


Summary of changes:
 src/app-layer.c             |    8 +-
 src/flow-hash.c             |   14 +-
 src/flow-manager.c          |   31 +-
 src/flow-private.h          |   19 -
 src/flow-util.h             |    3 +
 src/flow.c                  |   28 +-
 src/flow.h                  |   15 +-
 src/runmodes.c              |   46 +-
 src/runmodes.h              |    5 +
 src/stream-tcp-reassemble.c | 1841 +++++++++++++++++++------------------------
 src/stream-tcp-reassemble.h |    9 +-
 src/stream-tcp-util.c       |    7 +
 src/stream-tcp-util.h       |    2 +
 src/stream-tcp.c            |   56 +-
 src/stream-tcp.h            |    3 +
 src/util-file.c             |   35 +-
 suricata.yaml.in            |    6 +
 17 files changed, 982 insertions(+), 1146 deletions(-)


More information about the Oisf-devel mailing list