<div dir="ltr">I wouldn't advise you to do any buffering anyhow. But I am curious if you're deleting transactions once you're done with them. Because, if you're not, you may be allocating a lot of memory (all tx instances) on long-lived HTTP connections.<br>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 9, 2013 at 1:06 PM, Anoop Saldanha <span dir="ltr"><<a href="mailto:anoopsaldanha@gmail.com" target="_blank">anoopsaldanha@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Tue, Apr 9, 2013 at 2:36 PM, Victor Julien <<a href="mailto:victor@inliniac.net">victor@inliniac.net</a>> wrote:<br>
> (bad juju to brian and ivan for top posting and/or html emails! :)<br>
><br>
> On 04/09/2013 10:21 AM, Ivan Ristic wrote:<br>
>> On Mon, Apr 8, 2013 at 3:54 PM, Anoop Saldanha <<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a><br>
>> <mailto:<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a>>> wrote:<br>
>><br>
>> On Mon, Apr 8, 2013 at 7:50 PM, Brian Rectanus <<a href="mailto:brectanu@gmail.com">brectanu@gmail.com</a><br>
>> <mailto:<a href="mailto:brectanu@gmail.com">brectanu@gmail.com</a>>> wrote:<br>
>> ><br>
>> > On Mon, Apr 8, 2013 at 9:16 AM, Brian Rectanus <<a href="mailto:brectanu@gmail.com">brectanu@gmail.com</a><br>
>> <mailto:<a href="mailto:brectanu@gmail.com">brectanu@gmail.com</a>>> wrote:<br>
>> >><br>
>> >><br>
>> >> On Mon, Apr 8, 2013 at 8:47 AM, Anoop Saldanha<br>
>> <<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a> <mailto:<a href="mailto:anoopsaldanha@gmail.com">anoopsaldanha@gmail.com</a>>><br>
>> >> wrote:<br>
>> >>><br>
>> >>> On Mon, Apr 8, 2013 at 3:42 PM, Victor Julien<br>
>> <<a href="mailto:victor@inliniac.net">victor@inliniac.net</a> <mailto:<a href="mailto:victor@inliniac.net">victor@inliniac.net</a>>><br>
>> >>> wrote:<br>
>> >>> > (moving to oisf-devel)<br>
>> >>> ><br>
>> >>> > On 04/08/2013 06:17 AM, Anoop Saldanha wrote:<br>
>> >>> >>>> I recollect we introduced path and query double decoding<br>
>> through<br>
>> >>> >>>> configurable params, and also we had this thing with query<br>
>> >>> >>>> decoding(single level). Can you explain a bit what the<br>
>> status was<br>
>> >>> >>>> previously. Seeing related failed uts.<br>
>> >>> >>>><br>
>> >>> >>><br>
>> >>> >>> We run the path normalization on the query through our<br>
>> >>> >>> HTPCallbackRequestUriNormalizeQuery callback. Previously we used<br>
>> >>> >>> htp_decode_path_inplace to normalize the query (e.g. for<br>
>> >>> >>> uridecoding).<br>
>> >>> >>> However, this was causing issues (remember that pcre "bug" we<br>
>> >>> >>> discussed<br>
>> >>> >>> a while back, where http:// turned into http:/).<br>
>> >>> >>><br>
>> >>> >>> In libhtp I copied htp_decode_path_inplace to<br>
>> >>> >>> htp_decode_query_inplace<br>
>> >>> >>> and also copied the config params and cfg funcs:<br>
>> >>> >>><br>
>> >>> >>><br>
>> <a href="https://github.com/inliniac/suricata/commit/d41c762689a08e6814dc93e8bfebeceab97175c3" target="_blank">https://github.com/inliniac/suricata/commit/d41c762689a08e6814dc93e8bfebeceab97175c3</a><br>
>> >>> >>><br>
>> >>> >>> Hack of the 1st order, which is wrong in many ways. But it<br>
>> basically<br>
>> >>> >>> allowed me to make sure we don't normalize the query as if<br>
>> it's path,<br>
>> >>> >>> esp with turning ftp:// into ftp:/ and such.<br>
>> >>> >>><br>
>> >>> >>> For 0.5 integration I think we need a proper solution. The only<br>
>> >>> >>> reason I<br>
>> >>> >>> pushed my hack like this was that I knew in 0.5 we would<br>
>> make things<br>
>> >>> >>> right.<br>
>> >>> >>><br>
>> >>> >><br>
>> >>> >> I think if we still want to double decode, we still require<br>
>> all of<br>
>> >>> >> these above things from our bundled htp.<br>
>> >>> >><br>
>> >>> >> -----<br>
>> >>> >><br>
>> >>> >> In 0.5.x, tx->request_uri_normalized has been removed, and<br>
>> we'd now<br>
>> >>> >> have to use the REQUEST_URI hook. We'll have to carry out the<br>
>> >>> >> reconstruction ourselves, and store it ourselves in our HTPState.<br>
>> >>> >><br>
>> >>><br>
>> >>> What are your thoughts on this?<br>
>> >>><br>
>> >>> ><br>
>> >>> > IIRC there is some function in libhtp that does just the<br>
>> decoding of<br>
>> >>> > uriencoding and unicode. We should probably just use that on<br>
>> the query<br>
>> >>> > and do the full normalization on the path.<br>
>> >>> ><br>
>> >>> > As a side thought: I think it would be nice to store path and<br>
>> query<br>
>> >>> > separately so that we can add http_path and http_query<br>
>> keywords later<br>
>> >>> > on.<br>
>> >>> ><br>
>> >>><br>
>> >>> We'd pretty much extract it directly from parsed_uri. Will have to<br>
>> >>> check if we need the extract double decode phase we have<br>
>> currently in<br>
>> >>> our bundled htp, in which case we'd need to store them separately.<br>
>> >>><br>
>> >><br>
>> >> Yes, all the normalized components are in tx->parsed_uri. This<br>
>> is what is<br>
>> >> used in ironbee to expose all the various parts like<br>
>> tx->parsed_uri->path<br>
>> >> and tx->parsed_uri->query.<br>
>> >><br>
>> >> Also note that the hostname should now be obtained from<br>
>> >> tx->request_hostname in 0.5.<br>
>> >><br>
>> >> -B<br>
>> ><br>
>> ><br>
>> > FYI, for an example using libhtp 0.5 see ironbee code. This was all<br>
>> > recently updated for 0.5.<br>
>> ><br>
>> > <a href="https://github.com/ironbee/ironbee/blob/0.7.x/modules/modhtp.c" target="_blank">https://github.com/ironbee/ironbee/blob/0.7.x/modules/modhtp.c</a><br>
>> ><br>
>><br>
>> Will have a look. Thanks.<br>
>><br>
>> Previously we would use tx->connp->conn->transactions to access txs<br>
>> in the state. Now that htp_connp_t is an opaque pointer how do I<br>
>> access the txs? Tried locating helper functions to retrieve it, but I<br>
>> didn't find any.<br>
>><br>
>> It's an oversight that there isn't a helper function to retrieve<br>
>> transactions on a connections. I will add one tomorrow.<br>
>><br>
>> Having said that, what is your use case that you require to retrieve<br>
>> transactions? I thought your code was driven by the callbacks, which > all<br>
>> come with a tx instance (via connp)? For my education, can you explain<br>
>> how you process connection data?<br>
>><br>
>><br>
><br>
> One of the things that we don't do out of the callbacks is logging the<br>
> requests. This is one of the things we need access to the TX store for.<br>
><br>
<br>
</div></div>And to add to it, since we already have the txs stored in a list<br>
inside libhtp, re-buffering the txs would come as a redundant task,<br>
from where I see it.<br>
<span class="HOEnZb"><font color="#888888"><br>
--<br>
Anoop Saldanha<br>
</font></span><div class="HOEnZb"><div class="h5">_______________________________________________<br>
Suricata IDS Devel mailing list: <a href="mailto:oisf-devel@openinfosecfoundation.org">oisf-devel@openinfosecfoundation.org</a><br>
Site: <a href="http://suricata-ids.org" target="_blank">http://suricata-ids.org</a> | Participate: <a href="http://suricata-ids.org/participate/" target="_blank">http://suricata-ids.org/participate/</a><br>
List: <a href="https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel" target="_blank">https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel</a><br>
Redmine: <a href="https://redmine.openinfosecfoundation.org/" target="_blank">https://redmine.openinfosecfoundation.org/</a><br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Ivan Ristić<br>
</div>