It occurs to me that there's very little point in search engines
crawling certain pages of an OpenGuide - anything with "action=edit" or
"action=delete" in the URL, at the very least, has no real value to
someone searching for information. Unfortunately, unless I'm missing
something, robots.txt syntax doesn't allow for matching on anything
other than the start of the path component of a URL, which doesn't help
It's been suggested to me that ``<meta robots="noindex">''
tags in the
<head> of a page are effective here, but I'm not sure of the best way to
implement this for edit pages without implementing it for *all* pages,
which would be careless.
I'm thinking of something in header.tt conditional upon the requested
URI containing "action=edit" or "action=delete", but before I wander
learning how to talk template::toolkit, I'd be interested to hear better
suggestions -- and other values of action we might care about, I guess.
The other, more complex but possibly "better" alternative, would be for
someone to run with the idea mooted in this thread:
Then edit pages could be /edit/Node_Name, deletes /delete/Node_Name, and
so on. This makes setting up a suitable robots.txt very simple indeed,
though it makes setting up the Apache rewrite rules a) a requirement,
instead of just a nice thing, and b) more complex than at present.
Thoughts, comments, suggestions and the like all invited.
PGP fingerprint 3E85 0C7A FE11 42E9 A599 094D AE16 90F0 81AE 16FF, ID 81AE16FF
Fremen add life to spice!