[Oisf-devel] libhtp 0.5.x integration - bug 775

[Victor Julien]

(moving to oisf-devel)

On 04/08/2013 06:17 AM, Anoop Saldanha wrote:
>>> I recollect we introduced path and query double decoding through
>>> configurable params, and also we had this thing with query
>>> decoding(single level).  Can you explain a bit what the status was
>>> previously.  Seeing related failed uts.
>>>
>>
>> We run the path normalization on the query through our
>> HTPCallbackRequestUriNormalizeQuery callback. Previously we used
>> htp_decode_path_inplace to normalize the query (e.g. for uridecoding).
>> However, this was causing issues (remember that pcre "bug" we discussed
>> a while back, where http:// turned into http:/).
>>
>> In libhtp I copied htp_decode_path_inplace to htp_decode_query_inplace
>> and also copied the config params and cfg funcs:
>> https://github.com/inliniac/suricata/commit/d41c762689a08e6814dc93e8bfebeceab97175c3
>>
>> Hack of the 1st order, which is wrong in many ways. But it basically
>> allowed me to make sure we don't normalize the query as if it's path,
>> esp with turning ftp:// into ftp:/ and such.
>>
>> For 0.5 integration I think we need a proper solution. The only reason I
>> pushed my hack like this was that I knew in 0.5 we would make things right.
>>
> 
> I think if we still want to double decode, we still require all of
> these above things from our bundled htp.
> 
> -----
> 
> In 0.5.x, tx->request_uri_normalized has been removed, and we'd now
> have to use the REQUEST_URI hook.  We'll have to carry out the
> reconstruction ourselves, and store it ourselves in our HTPState.
> 

IIRC there is some function in libhtp that does just the decoding of
uriencoding and unicode. We should probably just use that on the query
and do the full normalization on the path.

As a side thought: I think it would be nice to store path and query
separately so that we can add http_path and http_query keywords later on.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------