[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-2.1beta3-36-g7426a9c

OISF Git noreply at openinfosecfoundation.org
Mon Feb 23 10:36:15 UTC 2015

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".

The branch, master has been updated
       via  7426a9c64504ac2731c36fae23f54a1a3639b9cc (commit)
       via  d78be75b9fff92484379d8c031948a3f0cbab986 (commit)
       via  6ad53627de604f7bd8ac9bfa8a20aec03c3b177c (commit)
       via  c88cbb39fec66faef315c5968d0cacd8e309a780 (commit)
       via  eaae008aeb68ac74468dad0fe473cf7cd04358ed (commit)
       via  11e3f25de3c910eb2b15b360bdc7594ee214a3fc (commit)
       via  29f70bad34dbf1bd55bd149158d23e9143c86ff5 (commit)
       via  2fb9611223824e16ae853a2aaac96183d147cfb0 (commit)
       via  cf58ecb084dec06a90a30185f825a38c648fa6d3 (commit)
       via  34e1de69702749c1523d4c73d70cf45ccdc83740 (commit)
       via  db9f12ee516bcb2c8513f1755bc278979d055813 (commit)
       via  d2eea7fd44b2cfcf7b740a249197dcabd88e8675 (commit)
       via  fb1b03c471700aa8de45b98210647aa91ec950ac (commit)
       via  3b05b735928d8bf2bc35097f6cc77a7f9009496e (commit)
       via  d834173bb8a7dc6bc73837fe720b3be3867f4194 (commit)
      from  bc2c7f462e79b5000636d5160301ee27dfff0f16 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit 7426a9c64504ac2731c36fae23f54a1a3639b9cc
Author: Victor Julien <victor at inliniac.net>
Date:   Wed Feb 18 08:50:01 2015 +0100

    flow: make TCP reuse handling in flow engine optional
    In case of autofp (or more general, when flow and stream engine
    run in different threads) the flow engine should not trigger a flow
    reuse as this can lead to race conditions between the flow and the
    stream engine.
    In such cases, the flow engine can be far ahead of the stream engine
    as packets are in a queue between the threads.
    Flow engine tags packet 10 as start of new flow. Flow is tagged as
    Stream engine evaluates packet 5 which belongs to the old flow. It
    rejects the flow as it's tagged 'reused'. Attaches packet 5 to the
    new flow which is wrong.
    This patch connects the flow engines handling of reuse cases to
    the runmode. It hooks into the RunmodeSetFlowStreamAsync() call to
    notify the flow engine that it shouldn't handle the reuse.

commit d78be75b9fff92484379d8c031948a3f0cbab986
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Feb 17 13:01:52 2015 +0100

    tcp reuse: don't double 'reuse'
    If the flow engine already reused a flow then the stream engine
    won't have to do the same.

commit 6ad53627de604f7bd8ac9bfa8a20aec03c3b177c
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Feb 17 12:47:01 2015 +0100

    flow: tag first packet in each direction
    Set a flowflag for the first packet in each direction:

commit c88cbb39fec66faef315c5968d0cacd8e309a780
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Feb 17 10:34:53 2015 +0100

    tcp reuse: enable stream handling based on runmode
    Add a way for runmodes to state that flow and stream run asynchorously.
    In the stream engine, enable the TCP reuse handling only if that flag
    is set.

commit eaae008aeb68ac74468dad0fe473cf7cd04358ed
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Feb 9 16:44:11 2015 +0100

    tcp reuse: support reuse on syn/ack
    Support TCP reuse on SYN/ACK. Only effective if midstream and/or
    async-oneside options are enabled.

commit 11e3f25de3c910eb2b15b360bdc7594ee214a3fc
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Feb 3 12:06:49 2015 +0100

    tcp reuse: unify autofp and single/workers check

commit 29f70bad34dbf1bd55bd149158d23e9143c86ff5
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Jan 22 17:30:10 2015 +0100

    tcp reuse: handle reuse in stream engine
    For the autofp case, handling TCP reuse in the flow engine didn't work.
    The problem is the mismatch between the moment the flow engine looks at
    packets and the stream, and the moment the stream engine runs. Flow engine
    is invoked in the packet capture thread(s), while the stream engine runs
    as part of the stream/detect thread(s). Because of the queues between
    those threads the flow manager may already inspect a new SYN while the
    stream engine still has to process the previous session.
    Moving the flow engine to the stream/detect thread(s) wasn't an option
    as the 'autofp' load balancing depends on the flow already being
    available in the packet.
    The solution here is to add a check for this condition to the stream
    engine. At this point the TCP state is up to date. If a TCP reuse case
    is encountered, this is the global logic:
    - detach packet for old flow
    - get a new flow and attach it to the packet
    - flag the old flow that it is now obsolete
    Additional logic makes sure that the packets already in the queue
    between the flow thread(s) and the stream thread are reassigned the
    new flow.
    Some special handling:
    Apply previous 'reuse' before checking for a new reuse. Otherwise we're
    tagging the wrong flow in some cases (multiple reuses in the same tuple).
    When in a flow/ssn reuse condition, properly remove the packet from
    the flow.
    Don't 'reuse' if packet is a SYN retransmission.
    The old flow is timed out normally by the flow manager.

commit 2fb9611223824e16ae853a2aaac96183d147cfb0
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Feb 2 17:14:49 2015 +0100

    flow: add util func to remove packet from flow
    Unsets the p::flowflags that were previously set.

commit cf58ecb084dec06a90a30185f825a38c648fa6d3
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Feb 2 15:46:24 2015 +0100

    flow: overwrite p::flowflags on first set

commit 34e1de69702749c1523d4c73d70cf45ccdc83740
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Feb 2 14:40:21 2015 +0100

    flow: move flow/packet updates into util func
    Move the code responsible for updating the flow and packet after
    a new packet has come in to a util func "FlowHandlePacketUpdate"

commit db9f12ee516bcb2c8513f1755bc278979d055813
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Dec 15 17:42:56 2014 +0100

    flow: add flow.tcp_reuse counter
    This replaces the tcp.reused_ssn counter. The flow engine now
    enforces the TCP flow reuse logic.
    The counter is incremented only when the flow is timed out, so
    after the "tcp closed" timeout expired for a flow.

commit d2eea7fd44b2cfcf7b740a249197dcabd88e8675
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Dec 15 16:01:28 2014 +0100

    tcp reuse: remove old code

commit fb1b03c471700aa8de45b98210647aa91ec950ac
Author: Victor Julien <victor at inliniac.net>
Date:   Mon Dec 15 15:06:53 2014 +0100

    flow: handle TCP session reuse in flow engine
    Until now, TCP session reuse was handled in the TCP stream engine.
    If the state was TCP_CLOSED, a new SYN packet was received and a few
    other conditions were met, the flow was 'reset' and reused for the
    'new' TCP session.
    There are a number of problems with this approach:
    - it breaks the normal flow lifecycle wrt timeout, detection, logging
    - new TCP sessions could come in on different threads due to mismatches
      in timeouts between suricata and flow balancing hw/nic/drivers
    - cleanup code was often causing problems
    - it complicated locking because of the possible thread mismatch
    This patch implements a different solution, where a new TCP session also
    gets a new flow. To do this 2 main things needed to be done:
    1. the flow engine needed to be aware of when the TCP reuse case was
    2. the flow engine needs to be able to 'skip' the old flow once it was
       replaced by a new one
    To handle (1), a new function TcpSessionPacketSsnReuse() is introduced
    to check for the TCP reuse conditions. It's called from 'FlowCompare()'
    for TCP packets / TCP flows that are candidates for reuse. FlowCompare
    returns FALSE for the 'old' flow in the case of TCP reuse.
    This in turn will lead to the flow engine not finding a flow for the TCP
    SYN packet, resulting in the creation of a new flow.
    To handle (2), FlowCompare flags the 'old' flow. This flag causes future
    FlowCompare calls to always return FALSE on it. In other words, the flow
    can't be found anymore. It can only be accessed by:
    1. existing packets with a reference to it
    2. flow timeout handling as this logic gets the flows from walking the
       hash directly
    3. flow timeout pseudo packets, as they are set up by (2)
    The old flow will time out normally, as governed by the "tcp closed"
    flow timeout setting. At timeout, the normal detection, logging and
    cleanup code will process it.
    The flagging of a flow making it 'unfindable' in the flow hash is a bit
    of a hack. The reason for this approach over for example putting the
    old flow into a forced timeout queue where it could be timed out, is
    that such a queue could easily become a contention point. The TCP
    session reuse case can easily be created by an attacker. In case of
    multiple packet handlers, this could lead to contention on such a flow
    timeout queue.

commit 3b05b735928d8bf2bc35097f6cc77a7f9009496e
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Feb 10 12:03:42 2015 +0100

    tcp midstream: fix window scaling
    If stream is picked by ACK, we can't know the wscale, so we assume it's
    set to max. Howver, we didn't apply this to the initial window size we

commit d834173bb8a7dc6bc73837fe720b3be3867f4194
Author: Victor Julien <victor at inliniac.net>
Date:   Tue Feb 17 12:08:20 2015 +0100

    detect-flow: use dedicated flags
    The flow keyword used flag names that were shared with the
    Packet::flowflags field. Some of the flags were'nt used by the packet
    though. This lead to waste of some 'flag space'.
    This patch defines dedicated flags for the flow keyword and removes
    the otherwise unused flags from the FLOW_PKT_* space.


Summary of changes:
 src/decode.h                 |    7 +
 src/detect-engine-analyzer.c |    3 +-
 src/detect-flow.c            |  174 ++++++++++---------
 src/detect-flow.h            |    7 +
 src/flow-hash.c              |  144 ++++++++++++++++
 src/flow-hash.h              |    2 +
 src/flow-manager.c           |   12 +-
 src/flow.c                   |   97 ++++++++---
 src/flow.h                   |   22 +--
 src/runmode-erf-file.c       |    1 +
 src/runmode-pcap-file.c      |    1 +
 src/stream-tcp.c             |  387 +++++++++++++++++++++++++++++++++++-------
 src/stream-tcp.h             |   14 ++
 src/util-runmodes.c          |   21 +++
 src/util-runmodes.h          |    2 +
 15 files changed, 706 insertions(+), 188 deletions(-)


More information about the Oisf-devel mailing list