<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style>
</head>
<body class='hmmessage'><div dir='ltr'>
Hi,<br><br>Thx, this is really helpful. I'm going to look at this. <br><br>What I need is to detect and block the illegitimate web aspirators. That means not blocking spiders from google for example. The apache module you spoke of might be a solution.<br><br>What exactly can be done with a reverse proxy regarding my problem ?<br><br>Thanks again.<br><br><div><hr id="stopSpelling">From: tcpandip@gmail.com<br>Date: Thu, 3 Nov 2011 09:11:04 -0400<br>Subject: Re: [Oisf-users] Web aspirator detection<br>To: amrith@hotmail.fr<br>CC: mcholste@gmail.com; oisf-users@openinfosecfoundation.org<br><br>Yea, I don't think IDS is the tool of choice for addressing/combating such activity. Perhaps there is another compelling piece of the puzzle we're missing.<div><br></div><div>What are the User-Agents?<br><div>Are they not respecting your robots.txt?</div>
<div>Firewall has already been mentioned (even iptables can handle).</div><div>If you're using Apache, ModSecurity could address. </div><div>Again, if you're using Apache, you might want to take a peek at mod_bandwidth and mod_limitipconn.</div>
<div>You might also want to check into the reverse proxy with Squid (or your proxy of choice with the capability).</div><div><br></div><div>And, yes, if you insist, an IDS signature could alert you given N connections over N timeframe. However, this can be very taxing depending on your parameters.<br>
<br><div class="ecxgmail_quote">On Thu, Nov 3, 2011 at 8:49 AM, Martin Holste <span dir="ltr"><<a href="mailto:mcholste@gmail.com">mcholste@gmail.com</a>></span> wrote:<br><blockquote class="ecxgmail_quote" style="border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
<div class="ecxim">> I'm looking for a way to detect web aspiration. I'm encountering a lot a<br>
> simultaneous connexions from single IPs, which are scrawling all our web<br>
> pages.<br>
<br>
</div>That is very normal. Web spiders from Google, Bing, Baidu, and<br>
thousands of others will continue to crawl pages, but it shouldn't<br>
cause a problem. Why do you want to detect the web crawls?<br>
<div class="ecxHOEnZb"><div class="h5">_______________________________________________<br>
Oisf-users mailing list<br>
<a href="mailto:Oisf-users@openinfosecfoundation.org">Oisf-users@openinfosecfoundation.org</a><br>
<a href="http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users" target="_blank">http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users</a><br>
</div></div></blockquote></div><br></div></div></div> </div></body>
</html>