> From: Jens Rösner [mailto:jens.roesner@xxxxxx]
> Subject: Re: robots.txt
> > or your indexing mech might loop on it, or crash the
> server. who knows.
> I have yet to find a site which forces wGet into a "loop" as you said.
> Others on the list probably can estimate the theoretical likelyhood of
> such events.
We do have some sites like that (dynamic pages with a possible infinite
depth), although not accessible to the whole internet.
So at least the probability is definitevely >0 :)
Heiko Herold
--
-- PREVINET S.p.A. Heiko.Herold@xxxxxxxxxxx
-- Via Ferretto, 1 ph x39-041-5907073
-- I-31021 Mogliano V.to (TV) fax x39-041-5907472
-- ITALY
|
