[Oisf-devel] libhtp 0.5.x integration - bug 775

[Brian Rectanus]

On Mon, Apr 8, 2013 at 8:47 AM, Anoop Saldanha <anoopsaldanha at gmail.com>wrote:

> On Mon, Apr 8, 2013 at 3:42 PM, Victor Julien <victor at inliniac.net> wrote:
> > (moving to oisf-devel)
> >
> > On 04/08/2013 06:17 AM, Anoop Saldanha wrote:
> >>>> I recollect we introduced path and query double decoding through
> >>>> configurable params, and also we had this thing with query
> >>>> decoding(single level).  Can you explain a bit what the status was
> >>>> previously.  Seeing related failed uts.
> >>>>
> >>>
> >>> We run the path normalization on the query through our
> >>> HTPCallbackRequestUriNormalizeQuery callback. Previously we used
> >>> htp_decode_path_inplace to normalize the query (e.g. for uridecoding).
> >>> However, this was causing issues (remember that pcre "bug" we discussed
> >>> a while back, where http:// turned into http:/).
> >>>
> >>> In libhtp I copied htp_decode_path_inplace to htp_decode_query_inplace
> >>> and also copied the config params and cfg funcs:
> >>>
> https://github.com/inliniac/suricata/commit/d41c762689a08e6814dc93e8bfebeceab97175c3
> >>>
> >>> Hack of the 1st order, which is wrong in many ways. But it basically
> >>> allowed me to make sure we don't normalize the query as if it's path,
> >>> esp with turning ftp:// into ftp:/ and such.
> >>>
> >>> For 0.5 integration I think we need a proper solution. The only reason
> I
> >>> pushed my hack like this was that I knew in 0.5 we would make things
> right.
> >>>
> >>
> >> I think if we still want to double decode, we still require all of
> >> these above things from our bundled htp.
> >>
> >> -----
> >>
> >> In 0.5.x, tx->request_uri_normalized has been removed, and we'd now
> >> have to use the REQUEST_URI hook.  We'll have to carry out the
> >> reconstruction ourselves, and store it ourselves in our HTPState.
> >>
>
> What are your thoughts on this?
>
> >
> > IIRC there is some function in libhtp that does just the decoding of
> > uriencoding and unicode. We should probably just use that on the query
> > and do the full normalization on the path.
> >
> > As a side thought: I think it would be nice to store path and query
> > separately so that we can add http_path and http_query keywords later on.
> >
>
> We'd pretty much extract it directly from parsed_uri.  Will have to
> check if we need the extract double decode phase we have currently in
> our bundled htp, in which case we'd need to store them separately.
>
>
Yes, all the normalized components are in tx->parsed_uri.  This is what is
used in ironbee to expose all the various parts like tx->parsed_uri->path
and tx->parsed_uri->query.

Also note that the hostname should now be obtained
from tx->request_hostname in 0.5.

-B
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-devel/attachments/20130408/97e98531/attachment-0002.html>