25 Jun
2008
25 Jun
'08
2:19 p.m.
On Wed 25 Jun 2008, David Sheldon dave@earth.li wrote:
In order to read the meta-headers the bot needs to make a request to the page. It wont index that page, or follow links from it though.
Hm, right, I see. I'd sort of assumed that it would only do that once, and then realise "OK, I don't need to check this page again". It's grabbed that particular one three times in the past week.
It might be worth putting some stuff in robots.txt to stop it, but I don't know if robots.txt can use the request parameters in patterns.
Bob says it can't.
Kake