[Oisf-users] Web aspirator detection

Martin Holste mcholste at gmail.com
Thu Nov 3 12:49:34 UTC 2011


> I'm looking for a way to detect web aspiration. I'm encountering a lot a
> simultaneous connexions from single IPs, which are scrawling all our web
> pages.

That is very normal.  Web spiders from Google, Bing, Baidu, and
thousands of others will continue to crawl pages, but it shouldn't
cause a problem.  Why do you want to detect the web crawls?



More information about the Oisf-users mailing list