Toolserver:OpenStreetMap server/Setup notes

This used to be on the Toolserver wiki but as of writing it's so fantastically slow as to be unusable. So I'm moving it here.

These are notes I (Ævar Arnfjörð Bjarmason) am keeping when setting up Cassini.

Software
This is the software running on Cassini

From apt
Non-standard software installed from the package system:


 * aptitude install screen pv bzip2
 * aptitude install subversion
 * aptitude build-dep mapnik
 * aptitude install python-cairo-dev libcairomm-1.0-dev # for mapnik
 * aptitude install iotop
 * aptitude install unzip # To get a stupd CPAN package that used a .zip file instead of gz/bzip2
 * aptitude install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj # for osm2pgsql
 * aptitude install postgresql-8.3-postgis
 * aptitude install apache2

From CPAN
Things installed from the CPAN. I'm using CPANPLUS (cpanplus) to install things. If this listing is incomplete /home/ravar/.cpanplus/install-logs/ should be the full listing.


 * WWW::Mechanize
 * HTML::TableParser::Grid
 * YAML::Syck
 * App::Ack

From source
Directories with stuff built from source, in /usr/local/src/

mapnik
ravar@cassini:/usr/local/src$ svn co http://svn.mapnik.org/trunk mapnik ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py configure ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py ravar@cassini:/usr/local/src/mapnik$ sudo python scons/scons.py install

osm
We're using various stuff from the OSM SVN, copied with:

ravar@cassini:/usr/local/src$ svn co http://svn.openstreetmap.org/ osm

osm2pgsql
ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ make ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ sudo make install

osm-tools
These are our tools for setting up or own rendering/stuff hosted in MediaWiki SVN:

ravar@cassini:/usr/local/src$ svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/osm-tools

Planet.osm
We have a copy of the Planet.osm file in /sql/planet.osm/

TODO: Keep it automatically up-to-date: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#--read-change-interval_.28--rci.29

PostGIS
I couldn't figure out quickly how to make a PostgreSQL user gis that I could log in as avar so I made avar a PostgreSQL superuser. TODO fix.

PostgreSQL has been setup to store its data on the /sql partition:

ravar@cassini:/sql$ sudo ack -a /sql /etc/postgresql /etc/postgresql/8.3/main/postgresql.conf 38:data_directory = '/sql/postgresql/8.3/main'         # use data in another directory

Imported databases
At first I wasn't using --slim, which is less usable, see: http://lists.openstreetmap.org/pipermail/talk/2009-July/038786.html

TODO: Re-import with intarray support to support OSC updating: see

With i18n support
$ createdb -E UTF8 -O gis gis $ createlang plpgsql gis $ psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql $ echo "ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;" | psql -d gis

$ perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl > /tmp/languages.yml $ perl /usr/local/src/osm-tools/osm2pgsql-style/create-wikimedia-style.pl --style /usr/local/src/osm/applications/utils/export/osm2pgsql/default.style --languages /tmp/languages.yml > /sql/planet.osm/wikimedia.style avar@cassini:/sql/planet.osm$ osm2pgsql --slim -d gis -U avar -W -S ./wikimedia.style planet-090715.osm

OSM-like
$ createdb -E UTF8 -O gis gis-osm-like $ createlang plpgsql gis-osm-like $ psql -d gis-osm-like -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql $ echo "ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;" | psql -d gis-osm-like

avar@cassini:/sql/planet.osm$ osm2pgsql --slim -d gis-osm-like -U avar -W -S /usr/local/src/osm/applications/utils/export/osm2pgsql/default.style planet-090715.osm

world_boundaries
As documented on the OSM wiki:

avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/world_boundaries-spherical.tgz avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/processed_p.zip avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/shoreline_300.tar.bz2

avar@cassini:/sql/world_boundaries/unpacked$ unzip ../packed/processed_p.zip Archive: ../packed/processed_p.zip inflating: coastlines/processed_p.dbf inflating: coastlines/processed_p.index inflating: coastlines/processed_p.shp inflating: coastlines/processed_p.shx avar@cassini:/sql/world_boundaries/unpacked$ tar xjvf ../packed/shoreline_300.tar.bz2 shoreline_300.dbf shoreline_300.index shoreline_300.shp shoreline_300.shx avar@cassini:/sql/world_boundaries/unpacked$ tar xzvf ../packed/world_boundaries-spherical.tgz world_boundaries/ world_boundaries/places.shx world_boundaries/world_boundaries_m.index world_boundaries/world_bnd_m.shx world_boundaries/builtup_area.shx world_boundaries/world_bnd_m.dbf world_boundaries/builtup_area.prj world_boundaries/places.shp world_boundaries/world_boundaries_m.shx world_boundaries/world_boundaries_m.shp world_boundaries/places.dbf world_boundaries/places.prj world_boundaries/builtup_area.dbf world_boundaries/world_bnd_m.shp world_boundaries/world_bnd_m.prj world_boundaries/world_boundaries_m.dbf world_boundaries/builtup_area.shp world_boundaries/world_boundaries_m.prj world_boundaries/world_bnd_m.index world_boundaries/builtup_area.index

avar@cassini:/sql/world_boundaries/unpacked$ mv coastlines/* world_boundaries/*. avar@cassini:/sql/world_boundaries/unpacked$ rmdir coastlines world_boundaries

avar@cassini:/sql/world_boundaries/unpacked$ ls -l total 534M -rw-r--r-- 1 avar users 3.4M 2007-03-10 10:05 builtup_area.dbf -rw-r--r-- 1 avar users 273K 2007-03-10 10:05 builtup_area.index -rw-r--r-- 1 avar users 513 2007-03-10 10:05 builtup_area.prj -rw-r--r-- 1 avar users 13M 2007-03-10 10:05 builtup_area.shp -rw-r--r-- 1 avar users 285K 2007-03-10 10:05 builtup_area.shx -rw-r--r-- 1 avar users 16K 2007-03-10 10:07 places.dbf -rw-r--r-- 1 avar users 333 2007-03-10 10:07 places.prj -rw-r--r-- 1 avar users 7.0K 2007-03-10 10:07 places.shp -rw-r--r-- 1 avar users 2.1K 2007-03-10 10:07 places.shx -rw-r--r-- 1 avar users 4.3M 2008-08-10 20:48 processed_p.dbf -rw-r--r-- 1 avar users 2.0M 2008-08-10 21:59 processed_p.index -rw-r--r-- 1 avar users 357M 2008-08-10 20:48 processed_p.shp -rw-r--r-- 1 avar users 3.4M 2008-08-10 20:48 processed_p.shx -rw-r--r-- 1 avar users 3.0M 2009-07-10 04:01 shoreline_300.dbf -rw-r--r-- 1 avar users 1.3M 2009-07-10 04:01 shoreline_300.index -rw-r--r-- 1 avar users 86M 2009-07-10 04:01 shoreline_300.shp -rw-r--r-- 1 avar users 2.0M 2009-07-10 04:01 shoreline_300.shx -rw-r--r-- 1 avar users 4.0M 2007-03-09 11:31 world_bnd_m.dbf -rw-r--r-- 1 avar users 246K 2007-09-09 19:28 world_bnd_m.index -rw-r--r-- 1 avar users 355 2007-03-09 11:31 world_bnd_m.prj -rw-r--r-- 1 avar users 47M 2007-03-09 11:32 world_bnd_m.shp -rw-r--r-- 1 avar users 244K 2007-03-09 11:32 world_bnd_m.shx -rw-r--r-- 1 avar users 659K 2008-03-30 21:42 world_boundaries_m.dbf -rw-r--r-- 1 avar users 85K 2008-03-30 21:48 world_boundaries_m.index -rw-r--r-- 1 avar users 355 2008-03-30 21:51 world_boundaries_m.prj -rw-r--r-- 1 avar users 6.4M 2008-03-30 21:42 world_boundaries_m.shp -rw-r--r-- 1 avar users 30K 2008-03-30 21:42 world_boundaries_m.shx