[Oisf-devel] Recommendation on accessing HTTP headers and payloads in lua at the same time

[Victor Julien]

On 05/24/2013 07:07 PM, Smutz, Charles wrote:
> 
> All, 
> 
> The lua functionality facilitates some pretty impressive capabilities. For example, the jar scanning the ET folk are doing demonstrates a big step in the realm of client object scanning in NIDS. I’m also a fan of the pcre based flowvar capture too, now available to lua.
> 
> In playing with this, a question we’ve had is: How can we access http header data at the same time we analyze the body data?
> 
> We’re largely interesting in farming this data out to an external scanner using something like lua bindings to zmq. However, whether the scanning is fully contained in the lua script or not, I think there is value in having access to this data (both header and body at the same time). For example, I could see some headers, such as the Server or Last-Modified headers, being relevant to analysis. Note that I’m most concerned about being able to analyze the request header with the request body and the response header with the response body, but it’s easy to come up with examples where the request header would be useful in analyzing the response body. Ex. jars which require URL query parameters.
> 
> Currently, if I understand correctly, you can have a script that operates on the request/response headers or body but not both at the same time. Is that correct? Are we missing something?
> 
> I see three possible ways to facilitate operating on the bodies and headers at the same time:
> 
> 1. Flowvar per metadata item: Use either lua script or pcre capture to set flowvars with the necessary header metadata, access these flowvars in the body analysis lua script. This seems possible, albeit rather inefficient and cumbersome.   
> 
> 2. External Storage: Have a lua script run on both headers and body. Extract the data necessary from both and analyze together externally. You need some way to link the headers and body. I’m not sure what the best way would be. Possibly through flowvar with a unique identifier set during header script?
> 
> 3. Access both header and body data directly, ex. using the needs declaration. As I understand, this isn’t possible today. However, it stands to reason that if you have access to the decoded http body, you’ve already parsed the headers (while the reverse is not always true) and should have a readily accessible reference to these headers also. Providing access to the headers at the same time the body is analyzed by lua would simplify this greatly.

The reason for this limitation is in the implementation details. We
inspect "content", "pcre" and all other related keywords in what we call
a "content inspection engine". This engine is passed a buffer and then
does it's magic on that buffer. It's mostly unaware of the type of
buffer, of the flow, etc. I've hooked the luajit support into this, so
it has the same limitation. The "needs" declaration merely tells
Suricata into what content inspection engine it should hook the script.

I agree it would be very nice to have all buffers available to the
script. Implementing this is probably non-trivial though. Feel free to
open a feature ticket.

-- 
---------------------------------------------
Victor Julien
http://www.inliniac.net/
PGP: http://www.inliniac.net/victorjulien.asc
---------------------------------------------