[ < ] | [ > ] | [ << ] | [ Up ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |
The most important thing about a program like this is to realise that if you set it up incorrectly and used it in the wrong way, you could upset a large number of people who have set up their web servers in the assumption that they would be used normally by human beings browsing through on Netscape.
Probably it's true that the only way forward is for every WWW site to begin to set up robot defences and detect when someone starts to download from at an unreasonable rate and then cut off the person doing the downloading. I suggest that you don't do this for at least two reasons.
There are probably many other reasons, but that's one for the good side in you and one for the selfish. What more do you need.
For suggestions about what constitutes `correct' behaviour, it's worth seeing the Robots World Wide Web page. http://info.webcrawler.com/mak/projects/robots/robots.html
There are a number of points which make LinkController relatively safe
as a link. These are all related to the design and limitations on
test-link
.
test-link
does not recurse. It only tests links that
are specifically listed in the schedule database.
The last limitation is inherited from the LWP::RobotUA
module and
the documentation for that covers the details of how it works.
test-link
trys to re-order testing of links as needed so that
a limit on the rate of visits to one site does not cause a limit on
overall testing speed.
[ << ] | [ >> ] | [Top] | [Contents] | [Index] | [ ? ] |