[NTLUG:Discuss] copying web documents

steve sjbaker1 at airmail.net
Thu May 18 17:55:49 CDT 2006


Stuart Johnston wrote:

> Apparently, wget respects the robots.txt file which causes problems with 
> this site.  But here's what I did.

wget only respects robots.txt when it's doing recursive loading.  For a 
single page - it's OK.

But you have the source code...so you know what to do!




More information about the Discuss mailing list