[Oisf-devel] transaction handling for unbalanced traffic

[Victor Julien]

I'm observing a case where the traffic profile leads to TX handling
problems. The traffic in question pushes tens of thousands of DNS
queries over the same flow (same 5 tuple), without ever receiving a
reply. This case leads to extreme slowdowns in the DNS handling.

This case isn't handled by AppLayerTransactionUpdateInspectId(). Even
though I flag the transactions as done within the DNS parser (reply
lost), the inspect id gets only updated for one direction, namely toserver.

Then, when it's time to clean up the completed transactions,
AppLayerTransactionsCleanup() takes the minimum of the toserver and
toclient inspect id. In this case it will always result in 0, no matter
what the toserver value is. Toclient is never updated, so always 0.

This leads to no transaction ever being cleared. Only at flow timeout,
the flow is freed and with it the transactions.

TCP won't have this problem, because even with data in one direction, we
still have packets (ACKs) in the other direction, updating the id's.
(although async will likely suffer from the same issues)


I'm not quite sure what the solution should be. In the short term I'm
just detecting this case in DNS, setting an event (flood detected) and
stop adding transactions. This at least stops the extreme slowdowns.


Maybe a good solution would be to have an app layer API call to indicate
that a TX is complete for both directions, which would also update the
inspect_id for the missing direction. In case of the DNS parser, I could
call this when I know the reply is lost.

Or we would have a per TX callback that is something like
AppLayerTxReplyLost(tx) that will return true in this case. Then
AppLayerTransactionUpdateInspectId() could take it into consideration to
update not just the toserver inspect_id, but also to the toclient
inspect_id. Then in the async tcp case, we'd automate this callback to
always return true.

Something like:
void AppLayerTransactionUpdateInspectId(Flow *f, uint8_t flags)
{
    uint8_t direction = (flags & STREAM_TOSERVER) ? 0 : 1;
    uint8_t opposite_direction =  (flags & STREAM_TOSERVER) ? 1 : 0;

    FLOWLOCK_WRLOCK(f);
    uint64_t total_txs = AppLayerGetTxCnt(f->alproto, f->alstate);
    uint64_t idx = AppLayerTransactionGetInspectId(f, flags);
    int state_done_progress =
AppLayerGetAlstateProgressCompletionStatus(f->alproto, direction);
    void *tx;
    int state_progress;

    for (; idx < total_txs; idx++) {
        SCLogDebug("idx %"PRIu64, idx);
        tx = AppLayerGetTx(f->alproto, f->alstate, idx);
        if (tx == NULL)
            continue;
        // new addition
        if (f->proto == IPPROTO_UDP && AppLayerTxReplyIsLost(tx))
            ((AppLayerParserStateStore
*)f->alparser)->inspect_id[opposite_direction] = idx;

        state_progress = AppLayerGetAlstateProgress(f->alproto, tx,
direction);
        SCLogDebug("need %d, have %d", state_done_progress, state_progress);
        if (state_progress >= state_done_progress)
            continue;
        else
            break;
    }
    ((AppLayerParserStateStore *)f->alparser)->inspect_id[direction] = idx;
    SCLogDebug("inspect_id now %"PRIu64, idx);
    FLOWLOCK_UNLOCK(f);

    return;
}

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------