Hi, I don't follow OG development but as host operator a few things are
coming up. The machine's load is gradually climbing over time and some
of that is OG.
Despite a relatively low hit rate on OG it is consuming quite a bit of
resource. If OG started taking off it would take the machine down.
First up: index.cgi requires 0.35s to perform a `perl -c` syntax check.
Any thoughts on putting OG on a mod_perl server? I have mod_perl running
here of course and we'd need to coordinate some apache.conf stuff.
Second: the supersearch.cgi gulps down CPU, often for seconds at a time.
It is a frequent resident of `top` output. This isn't really
acceptable. I'm going to request this feature be turned off unless an
effective optimisation plan or some other way to reduce its impact
here is constructed pretty soon. Sorry about this but it's encroaching
Third: I wonder if there's some way to instruct robots not to spider
parts of your wiki. This ought to speak for itself:
$ grep crawl /var/log/apache/london.openguides.org-access.log | grep 'action=edit' | wc -l
Finally: I posted about a DoS and was wondering what the status of a
solution was. http://openguides.org/mail/openguides-dev/2004-October/000542.html
Paul (any overbearing tone unintentional ;-)
Paul Makepeace .............................. http://paulm.com/inchoate/
"If my elbow was straight, then I'll show oyu mine!"
Some work's been done on this - see http://dev.openguides.org/ticket/81
- and I'm just looking for comments on this change really. PerfDave, do
can you explain you reason for wanting this? I think it'll help me
understand the value of having it if so. One thing would be for
translation purposes, I suppose?
Dominic Hargreaves | http://www.larted.org.uk/~dom/
PGP key 5178E2A5 from the.earth.li (keyserver,web,email)
I recently received a request from someone looking to set up the Open
Guide to Atlanta. I spent an hour last night doing the steps that are
needed to set up a new guide on Athena (the boston.og.org webserver):
* Create web directories with appropriate permissions (5 minutes --
* Copy over Boston web directory. This ended up including a couple
things that I didn't mean for it to, like extra files I had
accidentally stuck in the directory. Clean those up (10 minutes)
* Grab new gmaps key. (5 minutes)
* Add a temporary domain forward from atlanta.crschmidt.net -- then
realize that meant I had to get a new gmaps key. D'oh. (10 minutes)
* Modify all the "base" lat and long pairs to use ones that worked for
Atlanta, rather than ones that worked for Boston. Note that this is
probably not required if I ever get around to updating my templates
and configuration to the latest code, since I'm assuming Dom made
these configuration params. This took about 10 minutes.
* Remove all the Boston-specific stuff - T lines (both special icons
and the polylines), town boundary drawing, etc. This took the longest
-- about 15 minutes -- but now that it's done, I can simply copy out
the Atlanta templates instead of the Boston ones.
* Copied over and set up configuration files for stats (a la
http://boston.openguides.org/stats/) and webstats (a la
http://boston.openguides.org/webstats/). 10 minutes.
* Copied over and set up scripts for loading places from
atlanta.zami.com, added 43 places to the guide, checked it out,
called it done.
Total time: 1 hour.
Total time next time around: Probably closer to 30 minutes.
hex, when you get back, if you could please set up
atlanta.openguides.org to point to 22.214.171.124 or as a CNAME to Boston,
I'd appreciate it.
Next step: Setting up a couple more Guides, that people can claim from
me later if they want. I'll do the hosting and admin until someone else
comes along who wants to admin the site -- then they can either keep it
on my server, or they can have the data and files and move elsewhere.
The ones that I plan to set up at the moment are:
Because I know people who know those areas well enough to add some
content. I would also be happy to set up guides for any of the places
listed on http://zami.com/ -- because I have a spider which can crawl
the data out of there a bit more easily than I'd be able to get it
elsewhere otherwise. If anyone has any requests for these places, I'd
love to hear them, and set them up.
(Note that crawling this data in the US is considered legal -- at least
for the time being. "Collection" protection only comes into play when
there is some level of creativity involved in the data -- which yellow
page listings are not. The original court case regarding this actually
used white pages as an example, which is exactly the data I'm copying at
Also, thanks to rich_gibson of Geocoder.us, Athena now has
non-restricted access to Geocoder.us. Thanks a million for that, Rich!
Geocoder.us offers a ton of great commercial services, so if you're in
the US and wanting to do location lookups, he's a good person to talk
This is long and rambly. I'm going to stop talking, and get on to
working on setting these things up.
Per a request on the Milton Keynes guide, I took up the matter of
looking at making searches include more than just the text of the entry,
but instead also listing the metadata in the nodes. I successfully
completed this in the following manner:
* Modify the rename.pl in the dist to be more along the lines of:
You can see here that this indexes phone address city postcode
website fax summary, as well as categories and locales.
* Tested this to make sure the results worked. Number searching seems
broken, but other than that it's in good shape.
* Head in to try and make this change to the underlying guide code.
* Realize that searching is basically hardcoded into write_node, with
easy way around it. resign self to this fact and decide that
double-indexing is the way to proceed.
* Write a CGI::Wiki Plugin. Bang head against wall until realizing you
need to set @ISA yourself.
* Write reindexer similar to reindex.pl above.
* Find "write_node" in Openguides.pm. Place Reindexer plugin above
write_node (see code snippet at end of email).
* Test it out. Seems to work okay.
* Upload Indexer.pm code to http://crschmidt.net/openguides/Indexer.pm
* Post to mailing list.
I'm not sure if anyone else will find this interesting, but I certainly
did, so I figured I'd share the knowledge in case anyone else out there
my $indexer = CGI::Wiki::Plugin::Indexer->new;
$wiki->register_plugin( plugin => $indexer );
$indexer->reindex($node, $content, \%metadata);
my $written = $wiki->write_node($node, $content, $checksum, \%metadata );
Currently, when running OpenGuides under mod_perl, redirects do not
work, because the Apache $r object associated with the CGI object do not
exist. (You get an error message: Can't call method "send_cgi_header" on
an undefined value at (eval 64) line 62.)
This happens both after an edit, and when following redirects, and
possibly in other cases as well.
After some investigation last night, I tracked the source of this down:
currently, OpenGuides redirects use a class method of CGI:
However, this does not work. It seems that class methods in this case
never initialize the Apache request variable that is needed in order to
The solution for this, is to replace CGI->redirect calls with:
my $q = new CGI;
I have too little knowledge to be able to speak to whether this is a
fault within CGI or a fault within OpenGuides usage of it. However, this
change has allowed me to successfully use mod_perl on the Open Guide to
Boston in testing, and this fix or a similar fix should probably be
integrated into OpenGuides in either case, since upgrading the system
CGI.pm is a difficult task. The current debian packages for
stable include mod_perl 1.999.021, which is not compatible with CGI.pm
3.16, and upgrading mod_perl requires a full removal due to an API
rename (as described at http://perl.apache.org/docs/2.0/rename.html).
The code that needs to be changed is in OpenGuides.pm, in two functions:
A patch which makes these changes is attached.
I know Jo's on here anyway, but this has ramifications for the
OpenGuides project so it's worth considering.
----- Forwarded message from Jo Walsh <jo(a)frot.org> -----
Date: Tue, 14 Feb 2006 18:14:36 -0800
From: Jo Walsh <jo(a)frot.org>
Subject: [Openstreetmap] Please Help Petition for Public Access to Geodata
X-Spam-Status: No, score=-2.6 required=5.0 tests=BAYES_00 autolearn=ham
dear openstreetmap list,
Please sign the petition: http://petition.publicgeodata.org/
Yesterday the INSPIRE Directive on European "spatial data infrastructure"
began its second reading in the European Parliament. Within a few
weeks or months it may entrench a policy of charging European citizens
to access public information they've already paid to collect. Members of
the European Parliament think this an obscure and dull technical issue;
but free public access to geographic information is essential for economic
innovation and civic co-operation.
Benjamin Henrion and I are starting the above petition to MEPs to
reject the current draft of the INSPIRE directive. It's all rather
last-minute; we may have as little as a month before the Directive
gets through its second reading; so it's important we try and get as
many signatures as we can, as soon as we can, and get the word out.
People writing open source software, people who create a lot of GPS
information as a byproduct of being a hiker or cycle courier, people
who love cartography and just want to improve the representation of
the world around them, all over Europe, are getting together through
http://openstreetmap.org/ to create and share geographic information.
New data exchange standards, at both ad-hoc and industry level,
allow for more of the creation of maps and description of the
environment to be carried out by people who know the location well;
map data, census information and other essential data about how people
and places interact, which is used to manage civic society.
INSPIRE ignores all these trends. It is a narrow view that was formed
without a full survey of what's being done with spatial information -
especially with GPS and GSM positioning based media, new kinds of
transport logistics systems, and the upcoming GALILEO satellite
network. GALILEO will guarantee a reasonable quality signal for free
to all citizens. A Directive about access to state-collected public
geographic information, should guarantee freedom of access to data.
If you are in Europe, please help us raise awareness among MEPs that this
is a much bigger issue than it seems, and shouldn't be rubber-stamped!
Help us by signing http://petition.publicgeodata.org/ . Please tell
other interest groups that you're a part of - non-geo, or even non-geek
- and ask other European friends to help us by signing the petition.
Please feel free to forward this mail, or bits of it, onwards.
http://publicgeodata.org/ is a wiki for collaboration on research into
the issue, into some of the history and context around this
legislation, also providing a quick rough guide for focusing energy
when contacting your MEPs about this regressive, short-termist law.
If INSPIRE passes its second reading with this wording, all our in-car
navigation systems, mobile phones, local search services, will become
more expensive and less useful; INSPIRE threatens to set back economic
innovation in location aware technology in Europe by many years.
Openstreetmap mailing list
----- End forwarded message -----
Dominic Hargreaves | http://www.larted.org.uk/~dom/
PGP key 5178E2A5 from the.earth.li (keyserver,web,email)