9 Sep
2004
9 Sep
'04
4:02 p.m.
This one time, at band camp, James Green wrote:
It occurs to me that there's very little point in
search engines
crawling certain pages of an OpenGuide - anything with "action=edit" or
"action=delete" in the URL, at the very least, has no real value to
someone searching for information. Unfortunately, unless I'm missing
something, robots.txt syntax doesn't allow for matching on anything
other than the start of the path component of a URL, which doesn't help
us here.
Solution here would be to change the syntax of the action= URLs to be
something like:
/action=edit/MyWikiPage
Then you could put /action=edit/ in your robots.txt file.
This would probably, if you're not already doing it, involve Apache
rewriting which, if you're not already doing it, be a bit more effort
than is really warranted.
Using the meta tag doesn't remove the problem of the bandwidth used.
--
Rev Simon Rumble <simon(a)rumble.net>
www.rumble.net
"Those who know nothing of foreign languages know nothing
of their own."
- Johann Wolfgang von Goethe