WWW::RobotRules warning could be more helpful

If you spider several sites and one of them has a broken robots.txt file you
can't tell which one since the warning doesn't tell you.

Around line 73 of RobotRules.pm
change:
        warn "RobotRules: Disallow without preceding User-agent\n";
to
        # njh@xxxxxxxxxxxxxx: added $netloc
        warn "RobotRules: $netloc Disallow without preceding User-agent\n";

-Nigel

-- 
Nigel Horne. Arranger, Composer, Typesetter.
NJH Music, Barnsley, UK.  ICQ#20252325
njh@xxxxxxxxxxxxx http://www.bandsman.co.uk