[Oisf-devel] [COMMIT] OISF branch, master, updated. suricata-2.0-67-g4929c84

Tue Apr 22 11:19:24 UTC 2014

This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "OISF".

The branch, master has been updated
       via  4929c840cd2657763b812fb3e72b23b3a41b9d77 (commit)
       via  5db228b9f6ece9ace59b42431b68b641a495a3d1 (commit)
       via  539bf57a651d2278f6160a291be74940a5f861e3 (commit)
       via  3fa818d08731461e33779a48ac4657ac17bb68ea (commit)
       via  1bd189a07633aa92eb524c68c2873bfcef4e233e (commit)
       via  ed46fd715d6e5a2a01441efd661221a5c000cf12 (commit)
       via  165f129c61c74d18deff78449636c5880e997bcc (commit)
       via  32aafa6a485ceeac5eb7773dcbbc932837bbca80 (commit)
       via  44b8be3732b44f0d6d6ed807add7d78a556102c2 (commit)
       via  c1dcc7a26889a5a17043dcac52725a5fa692de81 (commit)
      from  59ccd7b62da0b18fa9d95f13358e4658bc8dabc5 (commit)

Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.

- Log -----------------------------------------------------------------
commit 4929c840cd2657763b812fb3e72b23b3a41b9d77
Author: Victor Julien <victor at inliniac.net>
Date:   Wed Apr 16 18:10:22 2014 +0200

    stream: update GAP detection

    Change GAP detection logic. If we encounter missing data before
    last_ack, we know we have missed data. The receiving host has ack'd
    it already, so a retransmission of the missing data is highly
    unlikely.

commit 5db228b9f6ece9ace59b42431b68b641a495a3d1
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Apr 11 17:31:46 2014 +0200

    stream: fix raw reassembly flag issue

    AppLayer reassembly correctly only flags a segment as done when it's
    completely used in reassembly. Raw reassembly could flag a partially
    used segment as complete as well. In this case the segment could be
    discarded early. Further reassembly would miss data, leading to a
    gap. Due to this, up to 'window size' bytes worth of segments could
    remain in the session for a long time, leading to memory resource
    pressure.

    This patch sets the flag correctly.

commit 539bf57a651d2278f6160a291be74940a5f861e3
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Apr 11 17:26:47 2014 +0200

    stream: improve StreamTcpPruneSession

    Check if a segment is done for the app-layer using
    StreamTcpAppLayerSegmentProcessed instead of the flag directly so the
    gap case works better.

commit 3fa818d08731461e33779a48ac4657ac17bb68ea
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Apr 11 17:24:12 2014 +0200

    stream: flags cleanup

    Stream flags are 16bit, but notation is still 8bit. Clean this up to
    avoid confusion.

commit 1bd189a07633aa92eb524c68c2873bfcef4e233e
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Apr 11 12:20:04 2014 +0200

    protocol detection: handle very unbalanced case

    Some traffic is very unbalanced. For example a 4 bytes request
    followed by 12mb of response data. If the 4 bytes don't lead to
    the protocol being detected, then app layer processing won't
    start, but it will not give up either. It will just wait for more
    data. This leads to piling up data on the opposite side.

    Observed:

    TS: 4 bytes. PP failed (bit set), PM has not given up (bit not set).
        This makes sense as max_depth is not yet reached.

    TC: 12mb. PP and PM failed (bits set).

    As ts-PM never gives up, we never consider proto detect complete,
    so all segments in either direction are still kept in the session.

    This patch adds a cutoff for this case:
    - IF for TS we have PP bit set, PM not set, AND
    - we have TC both bits set, AND
    - proto is unknown, AND
    - TC data is 100k already, THEN
    - give up on protocol detection.

    The same for the opposite direction.

commit ed46fd715d6e5a2a01441efd661221a5c000cf12
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Apr 11 10:11:04 2014 +0200

    stream: improve midstream reassembly gap detection

    The reassembly gap detection makes use of the window. However, in
    midstream mode the window size we use is unreliable, as we have to
    assume window scaling is in place. This patch improves midstream
    handling of those cases.

commit 165f129c61c74d18deff78449636c5880e997bcc
Author: Victor Julien <victor at inliniac.net>
Date:   Fri Apr 11 10:08:55 2014 +0200

    stream: detect data gap at stream start

    In midstream mode we may encounter a case where the data we is beyond
    the isn, but below last_ack. This means we're missing some data, that
    is already acked so it won't be retransmitted. Therefore, we can
    conclude it's a data gap.

commit 32aafa6a485ceeac5eb7773dcbbc932837bbca80
Author: Victor Julien <victor at inliniac.net>
Date:   Wed Jan 22 19:06:31 2014 +0100

    proto detect: add cutoff for unbalanced traffic

    If we're getting a lot of data in one direction and the proto for this
    direction is unknown, proto detect will hold up segments in the segment
    list in the stream. They are held so that if we detect the protocol on
    the opposing stream, we can still parse this side of the stream as well.
    However, some sessions are very unbalanced. FTP data channels, large
    PUT/POST request and many others, can lead to cases where we would have
    to store many megabytes worth of segments before we see the opposing
    stream. This leads to risks of resource starvation.

    In this patch, a cutoff point is enforced. If we've stored 100k in one
    direction and we've seen no data in the other direction, we give up.

    If we've given up, the applayer_proto_detection_skipped event is set.
        app-layer-event: applayer_proto_detection_skipped;

commit 44b8be3732b44f0d6d6ed807add7d78a556102c2
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Apr 10 17:59:18 2014 +0200

    protocol detection: midstream handling update

    If a TCP session is midstream, we may end up with a case where the
    start of an HTTP request is missing. We won't detect HTTP based on
    the request. However, the reply is fine, so we detect HTTP anyway.
    This leads to passing the incomplete request to the htp parser.

    This has been observed, where the http parser then saw many bogus
    requests in the incomplete data. This is not limited to HTTP.

    To counter this case, a midstream session MUST find it's protocol
    in the toserver direction. If not, we assume the start of the
    request/toserver is incomplete and no reliable detection and parsing
    is possible. So we give up.

commit c1dcc7a26889a5a17043dcac52725a5fa692de81
Author: Victor Julien <victor at inliniac.net>
Date:   Thu Apr 10 17:26:00 2014 +0200

    stream: fix midstream syn/ack setup

    If midstream is enabled and the first packet is the syn/ack packet from
    the 3whs, initialized server.last_ack to the packets seq.

    This fixes tracking the session.

-----------------------------------------------------------------------

Summary of changes:
 src/app-layer.c             |   91 +++++++++++++++++++++++++
 src/stream-tcp-private.h    |   18 ++---
 src/stream-tcp-reassemble.c |  153 ++++++++++++++++++++++++-------------------
 src/stream-tcp.c            |    2 +
 4 files changed, 187 insertions(+), 77 deletions(-)

hooks/post-receive
-- 
OISF