[Oisf-devel] Support for HTTP range requests

[Anoop Saldanha]

On Fri, Oct 9, 2015 at 3:54 PM, Victor Julien <victor at inliniac.net> wrote:
> On 08-10-15 21:02, Christian Kreibich wrote:
>>> We'll need to update libhtp a bit to start supporting this. I do think
>>> the 'reassembly' would better be done inside Suricata.
>>
>> Mhmm yeah, this seems a bit tricky. There could be various policies you
>> want to use to drive the reassembly. My intuition would be to have most
>> of it happen in libhtp since the content-ranges are an HTTP-internal
>> aspect that the library user may not care about at all.
>
> The thing that makes this tricky is storing the cross flow data in a
> thread safe and performant way. Libhtp currently just works on single
> flows, which fits nicely in Suricata's flow structure and leaves all the
> synchronisation issues to us.
>
> For byte-ranges we'll need cross flow storage and lookups.
>
> We could of course look at making the actual storage/lookup a set of
> callbacks that Suricata would register. But since we're the only serious
> (known & contributing) user of libhtp, I don't want to spend many extra
> cycles on making the lib nicer just for the sake of it. Of course I'd
> welcome contribs here (and elsewhere).
>
>>> We do need to think on how to do the data storage. For simple cases we
>>> may look at the IPPair data structure, but for sites with multiple hosts
>>> serving a domain we'd need something else. Perhaps a new data store
>>> searchable by hostname + url.
>>
>> In case it helps, RFC 7233 has this to say:
>>
>>   When a 206 response is generated, the server MUST generate the
>>   following header fields, in addition to those required above, if the
>>   field would have been sent in a 200 (OK) response to the same
>>   request: Date, Cache-Control, ETag, Expires, Content-Location, and
>>   Vary.
>>
>> So there are several ways to track the entity overall.
>
> That's useful, thanks!
>

The way I would look at this, libhtp is a http parser, and it would be
ideal if we update libhtp to extract range related data, and just
treat it as another body/file, but keep out supporting cross flow
range tracking, inside it.

I see the point in building this logic inside suricata then, but then
it gets tricky.  What sort of intelligence should we build into
suricata to track cross flow ranges.  How should we handle
overlapping.  These policies are better located outside suricata
inside another correlation engine, which can handle each of these
range based on case by case policy, and carry out reassembling the
whole file, as it sees fit.  Also if implemented within suricata, this
brings up the issue of how long to hold this data against DOS attacks,
etc.

-- 
-------------------------------
Anoop Saldanha
http://www.poona.me
-------------------------------