[Oisf-users] Suricata 1.4 http keywords in rule options, how does matching occur for http_header?

Fri Jan 25 19:53:42 UTC 2013

Sorry for the late response again, I had to rerun the tests,without the
patch, for the first case you explained so here is an archive for you and
anyone else who would like to look at the results. I'm mainly trying to
find out why wireshark with the filter I'm using is seeing 40 results while
the http_header and pcre in suricata only sees 36.

Turned off checksum-validation: no in suricata.yaml

Here's a summary again for the first case without the patch Anoop created
http-test.rules:
alert ip $HOME_NET any -> any any (msg:"http header rule fired"; content:"
businessweek.com"; http_header; sid:1;)
alert ip $HOME_NET any -> any any (msg:"pcre rule fired"; pcre:"/.*?
businessweek.com/H"; sid:2;)

wireshark filter:
http contains "businessweek.com" && ip.src == 192.168.32.136

wireshark had 40 matches
http rule had 36 alerts
pcre had 36 alerts

if you examine the packet number in the pcap file from wireshark, these
were the 4 that wireshark thought matched but I couldn't see
businessweek.com in any of the fields in them
663 -> This one had businessweek.com in the URI but not in any of the http
header fields, I'm not sure if this one should count or not
741
971
1043

Excluding those 4, everything seems to make sense. If anyone here is an
expert at wireshark, please take a look at the pcap file and explain what
other filter should I put in place to make it so the number of packets
matches the number of alerts found in suricata or why Wireshark sees more
than suricata, with the fact known that I'm looking at the entire http
header, not just the Host: field.

Now for version 2 where I would use the http header rule

alert ip $HOME_NET any -> any any (msg:"alert fired"; content:"Host:";
content:".businessweek.com"; within:40; http_header; sid:1;)

Suricata is complaining that it is an invalid siganture saying:

http_header seen with a distance or within without a previous http_header
content. Invalidating signature.

I modified the rule to look like this now and both new rules work as follows

alert ip $HOME_NET any -> any any (msg:"header rule fired"; content:"Host:"
http_header; content:"businessweek.com" http_header; within:40; sid:1;)
alert ip $HOME_NET any -> any any (msg:"pcre rule fired"; pcre:"/Host:\s.*?
businessweek.com/H"; sid:2;)

wireshark filter:
http.host contains "businessweek.com" && ip.src == 192.168.32.136

wireshark had 10 matches

fast.log was 20 lines long with
10 for http header
10 for pcre

So the second versions of rules work exactly the way I expected and this
definitely what I'm looking for. I will try out the patched version later
today and give you the results.

On Fri, Jan 25, 2013 at 8:19 AM, Anoop Saldanha <anoopsaldanha at gmail.com>wrote:

> Try this patch out(you can apply the patch using "git am -3 <patch>")
>
> It introduces a new keyword + pcre modifier that would inspect just
> the host header.
>
> The keyword being "http_host" and the pcre modifier being "W"
>
> You can now use it in a rule like this -
>
> alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
> pcre:"/\.businessweek.com/W"; sid:1;)
> alert ip $HOME_NET any -> any any (msg:"http header rule fired";
> content:".businessweek.com"; http_host; sid:2;)
>
> Let me know how it works with the above rules.
>
> On Fri, Jan 25, 2013 at 8:47 AM, Anoop Saldanha <anoopsaldanha at gmail.com>
> wrote:
> > On Thu, Jan 24, 2013 at 10:29 PM, Vincent Fang <vincent.y.fang at gmail.com>
> wrote:
> >> Here's the new results, I will run the tests again to see if it's
> consistent
> >> but using the wireshark filter
> >>
> >> http contains "businessweek.com"
> >>
> >> there were 75 matches
> >>
> >> and in the fast.log there were 138 total alerts from the two new rules
> you
> >> specified
> >> grep -c "http header" fast.log -> 69 lines
> >> grep -c "pcre version" fast.log -> 69 lines
> >>
> >> so they're both the same. Ran suricata in offline mode and the results
> were
> >> the same so that's good since they're consistent.
> >>
> >> Here's a copy of the two rules
> >>
> >> alert ip $HOME_NET any -> any any (msg:"pcre version rule fired";
> >> pcre:"/\s.*?\.businessweek.com/H"; sid:1;)
> >>
> >> alert ip $HOME_NET any -> any any (msg:"http header rule fired";
> >> content:".businessweek.com"; http_header; sid:2;)
> >>
> >> In the next few runs I also plan to change the protocol to http instead
> of
> >> ip, and I technically should get the same numbers yes?
> >>
> >
> > Yes, you should.
> >
> > Keep in mind that the above rules can also match on other headers
> > containing businessweek.com, for example the referer header.
> >
> >>
> >> On Thu, Jan 24, 2013 at 9:44 AM, Anoop Saldanha <
> anoopsaldanha at gmail.com>
> >> wrote:
> >>>
> >>> Sound good.  Will open a feature request for "http_host" keyword;
> >>>
> >>> On Thu, Jan 24, 2013 at 7:45 PM, Matt <matt at somedamn.com> wrote:
> >>> > I would find that useful, especially if it increases efficiency in
> the
> >>> > same
> >>> > way as http_user_agent.  Among other things, I use Suricata to match
> >>> > blacklists of known bad URLs, and all those rules include a content
> >>> > match
> >>> > for the HTTP Host.
> >>> >
> >>> > Matt
> >>> >
> >>> > On 1/24/2013 3:13 AM, Peter Manev wrote:
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Jan 24, 2013 at 9:11 AM, Anoop Saldanha
> >>> > <anoopsaldanha at gmail.com>
> >>> > wrote:
> >>> >>
> >>> >> On Thu, Jan 24, 2013 at 1:37 PM, Peter Manev <petermanev at gmail.com>
> >>> >> wrote:
> >>> >> >
> >>> >> >> However, any of the techniques mentioned above isn't a foolproof
> way
> >>> >> >> to match on the host header.  The right way would be to provide a
> >>> >> >> new
> >>> >> >> keyword called "http_host".
> >>> >> >>
> >>> >> > Anoop or Vincent would you please put in feature request for that?
> >>> >> >
> >>> >>
> >>> >> We should probably consult users/rule-writers if such a keyword
> would
> >>> >> be useful to them?
> >>> >>
> >>> >> --
> >>> >> Anoop Saldanha
> >>> >
> >>> > sure
> >>> >
> >>> >
> >>> > --
> >>> > Regards,
> >>> > Peter Manev
> >>> >
> >>> >
> >>> > _______________________________________________
> >>> > Suricata IDS Users mailing list:
> oisf-users at openinfosecfoundation.org
> >>> > Site: http://suricata-ids.org | Support:
> >>> > http://suricata-ids.org/support/
> >>> > List:
> >>> > https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>> > OISF: http://www.openinfosecfoundation.org/
> >>> >
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Anoop Saldanha
> >>> _______________________________________________
> >>> Suricata IDS Users mailing list: oisf-users at openinfosecfoundation.org
> >>> Site: http://suricata-ids.org | Support:
> http://suricata-ids.org/support/
> >>> List:
> https://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
> >>> OISF: http://www.openinfosecfoundation.org/
> >>
> >>
> >
> >
> >
> > --
> > Anoop Saldanha
>
>
>
> --
> Anoop Saldanha
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20130125/40ef0fe2/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: version1.7z
Type: application/octet-stream
Size: 306269 bytes
Desc: not available
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20130125/40ef0fe2/attachment-0002.obj>