[Oisf-users] Web aspirator detection

[Amrith Z]

Thx Kevin. Very helpful too. 
I've found another Apache module that might do the job :  mod_dosevasive 

But still, if I also want Suricata to alert this kind of connection, do I need to write a rule myself ?

Thx

Date: Thu, 3 Nov 2011 16:04:10 +0000
From: kevross33 at googlemail.com
To: oisf-users at openinfosecfoundation.org
Subject: Re: [Oisf-users] Web aspirator detection

Forgot to send response to all.

On 3 November 2011 16:03, Kevin Ross <kevross33 at googlemail.com> wrote:

It can be setup as a reverse proxy if you want or installed individually. There are books on modsecurity and stuff online about it. If you use the pfsense firewall (www.pfsense.org) it has a reverse proxy for modsecurity but I have not tried it as I use it more in a home environment. 



Modsecurity itself will detect all sorts of attacks and has sigs for specific stuff or attack types. You should look at the gootroot stuff (http://www.gotroot.com/Welcome) for it too as they have extra rules with free delayed non-subscriber release. There is also something you can install for a product they sell which you can try which has security modules, monitoring, GUI etc http://www.atomicorp.com/products.html



Network IDS can also detect bad stuff that it knows about so if we know something is bad we can detect it but modsecurity may be best for this. I would also look at Ossec for a host intrusion detection system which can run agents on *nix and windows systems and provides correlation to detect and block attacks (I am sure you can create rules and things for it if they don't exist to highlight suspicious things in your log files).


 

On 3 November 2011 15:44, Amrith Z <amrith at hotmail.fr> wrote:







Hi,

Thx, this is really helpful. I'm going to look at this. 

What I need is to detect and block the illegitimate web aspirators. That means not blocking spiders from google for example. The apache module you spoke of might be a solution.



What exactly can be done with a reverse proxy regarding my problem ?

Thanks again.

From: tcpandip at gmail.com

Date: Thu, 3 Nov 2011 09:11:04 -0400
Subject: Re: [Oisf-users] Web aspirator detection
To: amrith at hotmail.fr
CC: mcholste at gmail.com; oisf-users at openinfosecfoundation.org



Yea, I don't think IDS is the tool of choice for addressing/combating such activity. Perhaps there is another compelling piece of the puzzle we're missing.
What are the User-Agents?


Are they not respecting your robots.txt?

Firewall has already been mentioned (even iptables can handle).If you're using Apache, ModSecurity could address. Again, if you're using Apache, you might want to take a peek at mod_bandwidth and mod_limitipconn.



You might also want to check into the reverse proxy with Squid (or your proxy of choice with the capability).
And, yes, if you insist, an IDS signature could alert you given N connections over N timeframe. However, this can be very taxing depending on your parameters.





On Thu, Nov 3, 2011 at 8:49 AM, Martin Holste <mcholste at gmail.com> wrote:




> I'm looking for a way to detect web aspiration. I'm encountering a lot a

> simultaneous connexions from single IPs, which are scrawling all our web

> pages.



That is very normal.  Web spiders from Google, Bing, Baidu, and

thousands of others will continue to crawl pages, but it shouldn't

cause a problem.  Why do you want to detect the web crawls?

_______________________________________________

Oisf-users mailing list

Oisf-users at openinfosecfoundation.org

http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users


 		 	   		  

_______________________________________________

Oisf-users mailing list

Oisf-users at openinfosecfoundation.org

http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users







_______________________________________________
Oisf-users mailing list
Oisf-users at openinfosecfoundation.org
http://lists.openinfosecfoundation.org/mailman/listinfo/oisf-users 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openinfosecfoundation.org/pipermail/oisf-users/attachments/20111103/0bc22891/attachment-0002.html>