Yesterday AltaVista's spider (Scooter/3.3) crawled over 100,000 pages
at my site, http://www.isna.org (which actually only has 328 rows in
table nodes). At a rate of one per second. For nearly a whole day.
Sheesh!
The stupidest looking fetches are like this:
66.94.232.246 - - [23/Nov/2004:02:39:59 -0800] "GET
/drupal/new?from=297&sort=asc&order=title&
amp;from=0&sort=asc&
amp;amp;amp;amp;amp;order=type&
from=40&sort=asc&
amp;amp;amp;amp;amp;amp;amp;amp;order=title&
amp;amp;amp;amp;amp;from=20&
amp;amp;sort=asc&
order=title&
from=257&
sort=asc&
amp;order=type&
amp;amp;from=60&
amp;amp;amp;amp;sort=asc&
amp;amp;amp;amp;amp;amp;amp;order=title&
amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;amp;order=title HTTP/1.0"
200 29532 "-" "Scooter/3.3"
How to prevent it from doing this? Searching the web finds others (not
drupal users) also complaining about Scooter, eg
http://www.webmasterworld.com/forum1/2273.htm.
Thanks,
Cheryl
--
[ Drupal support list | http://list.drupal.org/ ]
|
Try Searching:
servers, voip, java, networking, microsoft ...
|
|
|
|