denied by robots.txt rules: msg#00301

nutch-user.lucene.apache.org

Subject:	denied by robots.txt rules

Hi
if a url is denied by denied once by robots.txt rules,is crawled again by
nutch?

--
View this message in context:
http://www.nabble.com/denied-by-robots.txt-rules-tp24750517p24750517.html
Sent from the Nutch - User mailing list archive at Nabble.com.

<Prev in Thread]	Current Thread	[Next in Thread>

Previous by Date:	denied by robots.txt rules, Saurabh Suman
Next by Date:	Re: Plugin development, Alexander Aristov
Previous by Thread:	denied by robots.txt rules, Saurabh Suman
Next by Thread:	Re: Hadoop java.io.IOException: Job failed! at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1232) while indexing., Filipe Antunes
Indexes:	[Date] [Thread] [Top] [All Lists]