[Oisf-devel] HTTP Log File

[Martin Holste]

Yes, it uses httpry as its backend for sniffing.  What's nice about
this is how ridiculously efficient httpry is.  It is much more
efficient than both Snort and Suricata's HTPlib for URL parsing
because it is lighter-weight, even though it will grab arbitrary
header fields.  This means it will offload quite a bit of CPU expense
from both Snort and Suricata.  On a high-speed network (> 100Mb/sec),
this is very noticeable.

What my wrapper does is matches requests with responses so the return
code, etc. is included and tacks on the country code.  It also breaks
up the domain name into subdomains so my fulltext search can use each
subdomain as a keyword (asdf.co.cc as well as co.cc).  I use the
syslog output for both IDS and httpry so that the URL's show up in my
syslog console right next to the Snort alert--no subquery required.

On Fri, Jul 8, 2011 at 9:41 AM, Brant Wells <bwells at tfc.edu> wrote:
> Hey Martin,
> Thanks for the reply!
> I already have the http.log file from Suricata with most of that info.  I
> was just wondering if anybody had built a parser for it, I guess.  It looks
> like httpry is made more for actually sniffing the traffic from the wire --
> am I correct?
> Thanks!
> ~Brant
>
>
> On Fri, Jul 8, 2011 at 9:27 AM, Martin Holste <mcholste at gmail.com> wrote:
>>
>> The easiest way to get them into a database would be to run my
>> httpry_logger script:
>>
>> http://code.google.com/p/enterprise-log-search-and-archive/downloads/detail?name=httpry_logger.pl
>> .  It has DB output as well as syslog and file outputs and adds GeoIP
>> tags to the URL entries.
>>
>> On Fri, Jul 8, 2011 at 12:15 AM, Brant Wells <bwells at tfc.edu> wrote:
>> > Hi All,
>> > I'm (finally) getting to dive back into getting my Suricata box going,
>> > and I
>> > have to say it is much easier now that I've done it a few times, lol.
>> > I have a couple of questions about the http.log file...
>> > 1) Is the output of that file compatible with utilities that analyze
>> > logs
>> > from Squid or what-not?
>> > 2) If the answer to #1 is no, then is there already a way to get the
>> > http.log file into a database?
>> > Just thought I'd ask...
>> > See Yas!
>> > ~Brant
>> >
>> >
>> > _______________________________________________
>> > Oisf-devel mailing list
>> > Oisf-devel at openinfosecfoundation.org
>> > http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-devel
>> >
>> >
>
>