Toolserver:OpenStreetMap server/Setup notes

These are notes I (Ævar Arnfjörð Bjarmason) am keeping when setting up Cassini.

Software
This is the software running on Cassini

From apt
Non-standard software installed from the package system:


 * aptitude install screen pv bzip2
 * aptitude install subversion
 * aptitude build-dep mapnik
 * aptitude install python-cairo-dev libcairomm-1.0-dev # for mapnik
 * aptitude install iotop
 * aptitude install unzip # To get a stupd CPAN package that used a .zip file instead of gz/bzip2
 * aptitude install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj # for osm2pgsql
 * aptitude install postgresql-8.3-postgis
 * aptitude install apache2
 * aptitude build-dep libapache2-mod-python # to get apache/apr dev headers for mod_tile
 * aptitude install strace
 * aptitude install gdb
 * aptitude install git-core
 * aptitude install ttf-unifont # for i18n support in rendering
 * aptitude install gdal-bin # for e.g. hillshading
 * aptitude install libgdal1-dev # for building Mapnik with gdal support

From CPAN
Things installed from the CPAN. I'm using CPANPLUS (cpanplus) to install things. If this listing is incomplete /home/ravar/.cpanplus/install-logs/ should be the full listing.


 * WWW::Mechanize
 * HTML::TableParser::Grid
 * YAML::Syck
 * App::Ack

From source
Directories with stuff built from source, in /usr/local/src/

mapnik
ravar@cassini:/usr/local/src$ svn co http://svn.mapnik.org/trunk mapnik

Make it build with gdal and other goodies:

ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py INPUT_PLUGINS='all'

Build it:

ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py configure ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py ravar@cassini:/usr/local/src/mapnik$ sudo python scons/scons.py install

By default mapnik can't find its .so:

ravar@cassini:/usr/local/src/mapnik$ python Python 2.5.2 (r252:60911, Jan 4 2009, 21:59:32) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mapnik Traceback (most recent call last): File " ", line 1, in  File "/usr/lib/python2.5/site-packages/mapnik/__init__.py", line 54, in     from _mapnik import * ImportError: libmapnik.so.0.6: cannot open shared object file: No such file or directory >>>

You need to add /usr/local/lib64 to /etc/ld.so.conf.d/x86_64-linux-gnu.conf

ravar@cassini:~$ diff -ru /etc/ld.so.conf.d/x86_64-linux-gnu.conf~ /etc/ld.so.conf.d/x86_64-linux-gnu.conf --- /etc/ld.so.conf.d/x86_64-linux-gnu.conf~   2009-07-21 14:54:36.000000000 +0000 +++ /etc/ld.so.conf.d/x86_64-linux-gnu.conf    2009-07-21 14:54:36.000000000 +0000 @@ -1,3 +1,4 @@ /lib/x86_64-linux-gnu /usr/lib/x86_64-linux-gnu +/usr/local/lib64
 * 1) Multiarch support

Then run:

sudo ldconfig

unifonts
aptitude install ttf-unifont

Make renderd / mapnik load it:

sudo ln -s /usr/share/fonts/truetype/unifont/unifont.ttf /usr/local/lib64/mapnik/fonts/unifont.ttf

osm
We're using various stuff from the OSM SVN, copied with:

ravar@cassini:/usr/local/src$ svn co http://svn.openstreetmap.org/ osm

osm2pgsql
ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ make ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ sudo make install

osm-tools
These are our tools for setting up or own rendering/stuff hosted in MediaWiki SVN:

ravar@cassini:/usr/local/src$ svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/osm-tools

cgi-bin/export
sudo aptitude install python-yaml sudo ln -s /usr/local/src/osm-tools/cgi-bin/export /usr/lib/cgi-bin/export sudo ln -s /usr/local/src/osm-tools/cgi-bin/export-config.yml /sql/etc/export-config.yml

renderd
Create the renderd.conf:


 * 1) perl create-renderd-conf.pl --languages /sql/misc-data/wikipedia-languages.yml > /etc/renderd.conf

Create the directories renderd emits to: avar@cassini:/sql/mod_tile$ mkdir $( perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl | perl -MYAML::Syck -E 'say join q[ ], sort keys %{ LoadFile(*STDIN) }' )

Planet.osm
We have a copy of the Planet.osm file in /sql/planet.osm/

TODO: Keep it automatically up-to-date: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#--read-change-interval_.28--rci.29

PostGIS
PostgreSQL has been setup to store its data on the /sql partition:

ravar@cassini:/sql$ sudo ack -a /sql /etc/postgresql /etc/postgresql/8.3/main/postgresql.conf 38:data_directory = '/sql/postgresql/8.3/main'         # use data in another directory

And for allowing password auth from localhost:

ravar@cassini:~$ sudo tail -n9 /etc/postgresql/8.3/main/pg_hba.conf
 * 1) "local" is for Unix domain socket connections only
 * 2) local  all         all                               ident sameuser
 * 3) IPv4 local connections:
 * 4) host   all         all         127.0.0.1/32          md5
 * 5) IPv6 local connections:
 * 6) host   all         all         ::1/128               md5

local  all         all                               trust host   all         all         127.0.0.1/32          trust

Imported databases
At first I wasn't using --slim, which is less usable, see: http://lists.openstreetmap.org/pipermail/talk/2009-July/038786.html

TODO: Re-import with intarray support to support OSC updating: see

With i18n support
createdb -E UTF8 -O gis gis createlang plpgsql gis psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql echo "ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;" | psql -d gis

Create language list & style:

perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl > /sql/misc-data/languages.yml perl /usr/local/src/osm-tools/osm2pgsql-style/create-wikimedia-style.pl --style /usr/local/src/osm/applications/utils/export/osm2pgsql/default.style --languages /sql/misc-data/languages.yml > /sql/planet.osm/wikimedia.style Import the planet:

osm2pgsql --slim -d gis -U avar -W -S ./wikimedia.style planet-090715.osm

world_boundaries
As documented on the OSM wiki:

avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/world_boundaries-spherical.tgz avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/processed_p.zip avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/shoreline_300.tar.bz2

avar@cassini:/sql/world_boundaries/unpacked$ unzip ../packed/processed_p.zip Archive: ../packed/processed_p.zip inflating: coastlines/processed_p.dbf inflating: coastlines/processed_p.index inflating: coastlines/processed_p.shp inflating: coastlines/processed_p.shx avar@cassini:/sql/world_boundaries/unpacked$ tar xjvf ../packed/shoreline_300.tar.bz2 shoreline_300.dbf shoreline_300.index shoreline_300.shp shoreline_300.shx avar@cassini:/sql/world_boundaries/unpacked$ tar xzvf ../packed/world_boundaries-spherical.tgz world_boundaries/ world_boundaries/places.shx world_boundaries/world_boundaries_m.index world_boundaries/world_bnd_m.shx world_boundaries/builtup_area.shx world_boundaries/world_bnd_m.dbf world_boundaries/builtup_area.prj world_boundaries/places.shp world_boundaries/world_boundaries_m.shx world_boundaries/world_boundaries_m.shp world_boundaries/places.dbf world_boundaries/places.prj world_boundaries/builtup_area.dbf world_boundaries/world_bnd_m.shp world_boundaries/world_bnd_m.prj world_boundaries/world_boundaries_m.dbf world_boundaries/builtup_area.shp world_boundaries/world_boundaries_m.prj world_boundaries/world_bnd_m.index world_boundaries/builtup_area.index

avar@cassini:/sql/world_boundaries/unpacked$ mv coastlines/* world_boundaries/*. avar@cassini:/sql/world_boundaries/unpacked$ rmdir coastlines world_boundaries

avar@cassini:/sql/world_boundaries/unpacked$ ls -l total 534M -rw-r--r-- 1 avar users 3.4M 2007-03-10 10:05 builtup_area.dbf -rw-r--r-- 1 avar users 273K 2007-03-10 10:05 builtup_area.index -rw-r--r-- 1 avar users 513 2007-03-10 10:05 builtup_area.prj -rw-r--r-- 1 avar users 13M 2007-03-10 10:05 builtup_area.shp -rw-r--r-- 1 avar users 285K 2007-03-10 10:05 builtup_area.shx -rw-r--r-- 1 avar users 16K 2007-03-10 10:07 places.dbf -rw-r--r-- 1 avar users 333 2007-03-10 10:07 places.prj -rw-r--r-- 1 avar users 7.0K 2007-03-10 10:07 places.shp -rw-r--r-- 1 avar users 2.1K 2007-03-10 10:07 places.shx -rw-r--r-- 1 avar users 4.3M 2008-08-10 20:48 processed_p.dbf -rw-r--r-- 1 avar users 2.0M 2008-08-10 21:59 processed_p.index -rw-r--r-- 1 avar users 357M 2008-08-10 20:48 processed_p.shp -rw-r--r-- 1 avar users 3.4M 2008-08-10 20:48 processed_p.shx -rw-r--r-- 1 avar users 3.0M 2009-07-10 04:01 shoreline_300.dbf -rw-r--r-- 1 avar users 1.3M 2009-07-10 04:01 shoreline_300.index -rw-r--r-- 1 avar users 86M 2009-07-10 04:01 shoreline_300.shp -rw-r--r-- 1 avar users 2.0M 2009-07-10 04:01 shoreline_300.shx -rw-r--r-- 1 avar users 4.0M 2007-03-09 11:31 world_bnd_m.dbf -rw-r--r-- 1 avar users 246K 2007-09-09 19:28 world_bnd_m.index -rw-r--r-- 1 avar users 355 2007-03-09 11:31 world_bnd_m.prj -rw-r--r-- 1 avar users 47M 2007-03-09 11:32 world_bnd_m.shp -rw-r--r-- 1 avar users 244K 2007-03-09 11:32 world_bnd_m.shx -rw-r--r-- 1 avar users 659K 2008-03-30 21:42 world_boundaries_m.dbf -rw-r--r-- 1 avar users 85K 2008-03-30 21:48 world_boundaries_m.index -rw-r--r-- 1 avar users 355 2008-03-30 21:51 world_boundaries_m.prj -rw-r--r-- 1 avar users 6.4M 2008-03-30 21:42 world_boundaries_m.shp -rw-r--r-- 1 avar users 30K 2008-03-30 21:42 world_boundaries_m.shx

/var/www/tile-browse
Create a HTML page to browse each of our 279 languages:

avar@cassini:/var/www/tile-browse$ for lang in $(perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl | perl -MYAML::Syck -E 'say join q[ ], sort keys %{ LoadFile(*STDIN) }'); do cp .browse-template.html browse-$lang.html && lang=$lang perl -pi -e 's[(?<=osm-like/)(\w+)][$ENV{lang}]' browse-$lang.html; done