<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'><div dir='ltr'>
Thx Kevin. Very helpful too. <br>I've found another Apache module that might do the job : mod_dosevasive <br><br>But still, if I also want Suricata to alert this kind of connection, do I need to write a rule myself ?<br><br>Thx<br><br><div><hr id="stopSpelling">Date: Thu, 3 Nov 2011 16:04:10 +0000<br>From: kevross33@googlemail.com<br>To: oisf-users@openinfosecfoundation.org<br>Subject: Re: [Oisf-users] Web aspirator detection<br><br>Forgot to send response to all.<br><br><div class="ecxgmail_quote">On 3 November 2011 16:03, Kevin Ross <span dir="ltr"><<a href="mailto:kevross33@googlemail.com">kevross33@googlemail.com</a>></span> wrote:<br><blockquote class="ecxgmail_quote" style="border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
It can be setup as a reverse proxy if you want or installed individually. There are books on modsecurity and stuff online about it. If you use the pfsense firewall (<a href="http://www.pfsense.org" target="_blank">www.pfsense.org</a>) it has a reverse proxy for modsecurity but I have not tried it as I use it more in a home environment. <br>
<br>Modsecurity itself will detect all sorts of attacks and has sigs for specific stuff or attack types. You should look at the gootroot stuff (<a href="http://www.gotroot.com/Welcome" target="_blank">http://www.gotroot.com/Welcome</a>) for it too as they have extra rules with free delayed non-subscriber release. There is also something you can install for a product they sell which you can try which has security modules, monitoring, GUI etc <a href="http://www.atomicorp.com/products.html" target="_blank">http://www.atomicorp.com/products.html</a><br>
<br>Network IDS can also detect bad stuff that it knows about so if we know something is bad we can detect it but modsecurity may be best for this. I would also look at Ossec for a host intrusion detection system which can run agents on *nix and windows systems and provides correlation to detect and block attacks (I am sure you can create rules and things for it if they don't exist to highlight suspicious things in your log files).<div class="ecxHOEnZb">
<div class="h5"><br>
<br><br><div class="ecxgmail_quote">On 3 November 2011 15:44, Amrith Z <span dir="ltr"><<a href="mailto:amrith@hotmail.fr">amrith@hotmail.fr</a>></span> wrote:<br><blockquote class="ecxgmail_quote" style="border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div><div dir="ltr">
Hi,<br><br>Thx, this is really helpful. I'm going to look at this. <br><br>What I need is to detect and block the illegitimate web aspirators. That means not blocking spiders from google for example. The apache module you spoke of might be a solution.<br>
<br>What exactly can be done with a reverse proxy regarding my problem ?<br><br>Thanks again.<br><div><br></div><div><div><hr>From: <a href="mailto:tcpandip@gmail.com">tcpandip@gmail.com</a><br>
Date: Thu, 3 Nov 2011 09:11:04 -0400<br>Subject: Re: [Oisf-users] Web aspirator detection<br>To: <a href="mailto:amrith@hotmail.fr">amrith@hotmail.fr</a><br>CC: <a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>; <a href="mailto:oisf-users@openinfosecfoundation.org">oisf-users@openinfosecfoundation.org</a></div>
<div><div><br><br>Yea, I don't think IDS is the tool of choice for addressing/combating such activity. Perhaps there is another compelling piece of the puzzle we're missing.<div><br></div><div>What are the User-Agents?<br>
<div>Are they not respecting your robots.txt?</div>
<div>Firewall has already been mentioned (even iptables can handle).</div><div>If you're using Apache, ModSecurity could address. </div><div>Again, if you're using Apache, you might want to take a peek at mod_bandwidth and mod_limitipconn.</div>
<div>You might also want to check into the reverse proxy with Squid (or your proxy of choice with the capability).</div><div><br></div><div>And, yes, if you insist, an IDS signature could alert you given N connections over N timeframe. However, this can be very taxing depending on your parameters.<br>
<br><div>On Thu, Nov 3, 2011 at 8:49 AM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span> wrote:<br><blockquote style="padding-left: 1ex;">
<div>> I'm looking for a way to detect web aspiration. I'm encountering a lot a<br>
> simultaneous connexions from single IPs, which are scrawling all our web<br>
> pages.<br>
<br>
</div>That is very normal. Web spiders from Google, Bing, Baidu, and<br>
thousands of others will continue to crawl pages, but it shouldn't<br>
cause a problem. Why do you want to detect the web crawls?<br>
<div><div>_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
</div></div></blockquote></div><br></div></div></div></div></div> </div></div>
<br>_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
<br></blockquote></div><br>
</div></div></blockquote></div><br>
<br>_______________________________________________
Oisf-users mailing list
Oisf-users@openinfosecfoundation.org
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</div> </div></body>
</html>