[Oisf-users] Web aspirator detection

[Kevin Ross]

Forgot to send response to all.

On 3 November 2011 16:03, Kevin Ross <kevross33 at googlemail.com> wrote:

> It can be setup as a reverse proxy if you want or installed individually.
> There are books on modsecurity and stuff online about it. If you use the
> pfsense firewall (www.pfsense.org) it has a reverse proxy for modsecurity
> but I have not tried it as I use it more in a home environment.
>
> Modsecurity itself will detect all sorts of attacks and has sigs for
> specific stuff or attack types. You should look at the gootroot stuff (
> http://www.gotroot.com/Welcome) for it too as they have extra rules with
> free delayed non-subscriber release. There is also something you can
> install for a product they sell which you can try which has security
> modules, monitoring, GUI etc http://www.atomicorp.com/products.html
>
> Network IDS can also detect bad stuff that it knows about so if we know
> something is bad we can detect it but modsecurity may be best for this. I
> would also look at Ossec for a host intrusion detection system which can
> run agents on *nix and windows systems and provides correlation to detect
> and block attacks (I am sure you can create rules and things for it if they
> don't exist to highlight suspicious things in your log files).
>
>
>
> On 3 November 2011 15:44, Amrith Z <amrith at hotmail.fr> wrote:
>
>>  Hi,
>>
>> Thx, this is really helpful. I'm going to look at this.
>>
>> What I need is to detect and block the illegitimate web aspirators. That
>> means not blocking spiders from google for example. The apache module you
>> spoke of might be a solution.
>>
>> What exactly can be done with a reverse proxy regarding my problem ?
>>
>> Thanks again.
>>
>> ------------------------------
>> From: tcpandip at gmail.com
>> Date: Thu, 3 Nov 2011 09:11:04 -0400
>> Subject: Re: [Oisf-users] Web aspirator detection
>> To: amrith at hotmail.fr
>> CC: mcholste at gmail.com; oisf-users at openinfosecfoundation.org
>>
>>
>> Yea, I don't think IDS is the tool of choice for addressing/combating
>> such activity. Perhaps there is another compelling piece of the puzzle
>> we're missing.
>>
>> What are the User-Agents?
>> Are they not respecting your robots.txt?
>> Firewall has already been mentioned (even iptables can handle).
>> If you're using Apache, ModSecurity could address.
>> Again, if you're using Apache, you might want to take a peek
>> at mod_bandwidth and mod_limitipconn.
>> You might also want to check into the reverse proxy with Squid (or your
>> proxy of choice with the capability).
>>
>> And, yes, if you insist, an IDS signature could alert you given N
>> connections over N timeframe. However, this can be very taxing depending on
>> your parameters.
>>
>> On Thu, Nov 3, 2011 at 8:49 AM, Martin Holste <mcholste at gmail.com> wrote:
>>
>> > I'm looking for a way to detect web aspiration. I'm encountering a lot a
>> > simultaneous connexions from single IPs, which are scrawling all our web
>> > pages.
>>
>> That is very normal.  Web spiders from Google, Bing, Baidu, and
>> thousands of others will continue to crawl pages, but it shouldn't
>> cause a problem.  Why do you want to detect the web crawls?
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>>
>>
>> _______________________________________________
>> Oisf-users mailing list
>> Oisf-users at openinfosecfoundation.org
>> http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20111103/53d90564/attachment-0002.html>