Re: WGET Syntax for a URL with cgi scripts.

Fred Holmes wrote:
> The following fails, with error messages that don't make sense to me.
>
> WGET -r -l2 -k -K -p -np -nd 
> "http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST&"; 
>
>
> The URL is a valid one.  (It's all one line, wrapped as you may receive 
> it.)  The URL works when copy/pasted into the address box of a browser.
>
> It's the Table of Contents page of an html version of House Rept. 
> 107-791.  HTML is much easier to read than the .pdf version (which can 
> be downloaded as a single file) or the ASCII version (which can also be 
> downloaded as a single file.
>
> I want to "snag" the entire document, which consists of "chapters" 
> linked to the headings in the Table of Contents document.
>
> It fails with different error messages depending upon whether I put the 
> URL in quotation marks (as above) or not.

Which version of wget do you use? Would please you include the debug 
output with the error message if you are running the latest version 
(1.8.2). As you can see I have no problem with this page.
Yes, you should use quotes to protect the '?' and the '&' from your shell.

$ wget -d -O content.html 
"http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST";
DEBUG output created by Wget 1.8.2 on cygwin.

--21:30:29-- 
http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST
           => `content.html'
Resolving thomas.loc.gov... done.
Caching thomas.loc.gov => 140.147.249.9
Connecting to thomas.loc.gov[140.147.249.9]:80... connected.
Created socket 4.
Releasing 0x10029388 (new refcount 1).
---request begin---
HEAD 
/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST 
HTTP/1.0
User-Agent: Wget/1.8.2
Host: thomas.loc.gov
Accept: */*
Connection: Keep-Alive


---request end---
HTTP request sent, awaiting response... HTTP/1.1 200 OK
Date: Wed, 08 Jan 2003 20:30:30 GMT
Server: Web
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html


Length: unspecified [text/html]
Closing fd 4
Last-modified header missing -- time-stamps turned off.
--21:30:30-- 
http://thomas.loc.gov/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndict=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST
           => `content.html'
Found thomas.loc.gov in host_name_addresses_map (0x10029388)
Connecting to thomas.loc.gov[140.147.249.9]:80... connected.
Created socket 4.
Releasing 0x10029388 (new refcount 1).
---request begin---
GET 
/cgi-bin/cpquery/0?r_t=a&r_n=107-791&submit=Search&variant=y&acronym=n&syndi
ct=n&spq=n&sort_by_docid=n&d_q=on&dbname=cp107&maxdocs=100&report=hr791.107&xform_type=3&sel=TOCLIST 
HTTP/1.0
User-Agent: Wget/1.8.2
Host: thomas.loc.gov
Accept: */*
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... HTTP/1.1 200 OK
Date: Wed, 08 Jan 2003 20:30:31 GMT
Server: Web
Connection: close
Content-Type: text/html


Length: unspecified [text/html]

    [    <=>                              ] 33,466        49.67K/s 


Closing fd 4
21:30:31 (49.67 KB/s) - `content.html' saved [33466]



> I haven't the foggiest idea how to obtain a "direct" address/URL for the 
> document (Table of Contents) or for the links.
>
> If WGET can't handle this is there any other application that will work 
> here?
>
> Thanks,
>
> Fred Holmes


--
Med venlig hilsen / Kind regards

Hack Kampbjørn