Choosing A Webhost:
A web hosting service is a type of Internet hosting service that allows individuals and organizations to provide their own website accessible via the World Wide Web. Web hosts are companies that provide space on a server they own for use by their clients as well as providing Internet connectivity, typically in a data center. Web hosts can also provide data center space and connectivity to the Internet for servers they do not own to be located in their data center, called colocation. more...

RE: robots.txt takes precedence over -p: msg#00068

web.wget.general

Subject:	RE: robots.txt takes precedence over -p

Thomas Boerner wrote:

> Is this behaviour: "robots.txt takes precedence over -p" a bug or
> a feature?

It is a feature. If you want to ignore robots.txt, use this command line:

wget -p -k www.heise.de/index.html -e robots=off

Tony

<Prev in Thread]	Current Thread	[Next in Thread>

Previous by Date:	robots.txt takes precedence over -p, Thomas Boerner
Next by Date:	Translating wget: reduntant "done" translations, Petri T. Koistinen
Previous by Thread:	robots.txt takes precedence over -p, Thomas Boerner
Next by Thread:	Translating wget: reduntant "done" translations, Petri T. Koistinen
Indexes:	[Date] [Thread] [Top] [All Lists]

Recently Viewed:
krysalis.sandbo... web.zope.zwiki/... gnome.apps.gnum... xfree86.newbie/... editors.vim/200... mozilla.enigmai... boot-loaders.gr... network.vnc.ult... redhat.release.... java.geronimo.u... os.netbsd.devel... horde.wicked/20... linux.lsb.discu... ietf.ips/2005-0... alsa.devel/2002... user-groups.lin... package-managem... debian.devel.da... security.cyrus.... video.gstreamer...

Home | blog view | USPTO Patent Archive | advertise | OSDir is an inevitable website. super tiny logo

Free Magazines

Cisco News
Receive a free quarterly e-newsletter with exclusive articles on how Cisco IT uses its own products and solutions to enable the business.
subscribe

Systems Management News, the newspaper for IT systems administration and data center managers! Each issue of Systems Management News is chock-full of news and analysis to help you understand what's happening in your field.
subscribe

The Enterprise Newsweekly eWeek is the essential technology information source for builders of e-business.
subscribe

Oracle Magazine Oracle Magazine contains technology strategy articles, sample code, tips, Oracle and partner news, how to articles for developers and DBAs, and more. Oracle (NASDAQ: ORCL) is the world's largest enterprise software company.
subscribe

Total Telecom Total Telecom is "The Economist of the communications industry".
subscribe

Navigation