Toolserver:OpenStreetMap server/Setup notes

From mediawiki.org

This page was moved from the Toolserver wiki.
Toolserver has been replaced by Toolforge. As such, the instructions here may no longer work, but may still be of historical interest.
Please help by updating examples, links, template links, etc. If a page is still relevant, move it to a normal title and leave a redirect.

Comment
Cassini is no longer in use and its successor ptolemy is setup differently. Haggi 19:04, 12 August 2011 (UTC)

These are notes I (Ævar Arnfjörð Bjarmason) am keeping when setting up Cassini.

Software[edit]

This is the software running on Cassini

From apt[edit]

Non-standard software installed from the package system:

  • aptitude install screen pv bzip2
  • aptitude install subversion
  • aptitude build-dep mapnik
  • aptitude install python-cairo-dev libcairomm-1.0-dev # for mapnik
  • aptitude install iotop
  • aptitude install unzip # To get a stupd CPAN package that used a .zip file instead of gz/bzip2
  • aptitude install build-essential libxml2-dev libgeos-dev libpq-dev libbz2-dev proj # for osm2pgsql
  • aptitude install postgresql-8.3-postgis
  • aptitude install apache2
  • aptitude build-dep libapache2-mod-python # to get apache/apr dev headers for mod_tile
  • aptitude install strace
  • aptitude install gdb
  • aptitude install git-core
  • aptitude install ttf-unifont # for i18n support in rendering
  • aptitude install gdal-bin # for e.g. hillshading
  • aptitude install libgdal1-dev # for building Mapnik with gdal support

From CPAN[edit]

Things installed from the CPAN. I'm using CPANPLUS (cpanplus) to install things. If this listing is incomplete /home/ravar/.cpanplus/install-logs/ should be the full listing.

  • WWW::Mechanize
  • HTML::TableParser::Grid
  • YAML::Syck
  • App::Ack

From source[edit]

Directories with stuff built from source, in /usr/local/src/

mapnik[edit]

ravar@cassini:/usr/local/src$ svn co http://svn.mapnik.org/trunk mapnik

Make it build with gdal and other goodies:

ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py INPUT_PLUGINS='all'

Build it:

ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py configure
ravar@cassini:/usr/local/src/mapnik$ python scons/scons.py
ravar@cassini:/usr/local/src/mapnik$ sudo python scons/scons.py install

By default mapnik can't find its .so:

ravar@cassini:/usr/local/src/mapnik$ python
Python 2.5.2 (r252:60911, Jan  4 2009, 21:59:32)
[GCC 4.3.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mapnik
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python2.5/site-packages/mapnik/__init__.py", line 54, in <module>
    from _mapnik import *
ImportError: libmapnik.so.0.6: cannot open shared object file: No such file or directory
>>>

You need to add /usr/local/lib64 to /etc/ld.so.conf.d/x86_64-linux-gnu.conf

ravar@cassini:~$ diff -ru /etc/ld.so.conf.d/x86_64-linux-gnu.conf~ /etc/ld.so.conf.d/x86_64-linux-gnu.conf
--- /etc/ld.so.conf.d/x86_64-linux-gnu.conf~    2009-07-21 14:54:36.000000000 +0000
+++ /etc/ld.so.conf.d/x86_64-linux-gnu.conf     2009-07-21 14:54:36.000000000 +0000
@@ -1,3 +1,4 @@
 # Multiarch support
 /lib/x86_64-linux-gnu
 /usr/lib/x86_64-linux-gnu
+/usr/local/lib64

Then run:

sudo ldconfig
unifonts[edit]
aptitude install ttf-unifont

Make renderd / mapnik load it:

sudo ln -s /usr/share/fonts/truetype/unifont/unifont.ttf /usr/local/lib64/mapnik/fonts/unifont.ttf

osm[edit]

We're using various stuff from the OSM SVN, copied with:

ravar@cassini:/usr/local/src$ svn co http://svn.openstreetmap.org/ osm
osm2pgsql[edit]
ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ make
ravar@cassini:/usr/local/src/osm/applications/utils/export/osm2pgsql$ sudo make install

osm-tools[edit]

These are our tools for setting up or own rendering/stuff hosted in MediaWiki SVN:

ravar@cassini:/usr/local/src$ svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/tools/osm-tools
cgi-bin/export[edit]
sudo aptitude install python-yaml
sudo ln -s /usr/local/src/osm-tools/cgi-bin/export /usr/lib/cgi-bin/export
sudo ln -s /usr/local/src/osm-tools/cgi-bin/export-config.yml /sql/etc/export-config.yml

renderd[edit]

Create the renderd.conf:

# perl create-renderd-conf.pl --languages /sql/misc-data/wikipedia-languages.yml > /etc/renderd.conf

Create the directories renderd emits to:

avar@cassini:/sql/mod_tile$ mkdir $( perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl | perl -MYAML::Syck -E 'say join q[ ], sort keys %{ LoadFile(*STDIN) }' )

Data[edit]

Planet.osm[edit]

We have a copy of the Planet.osm file in /sql/planet.osm/

TODO: Keep it automatically up-to-date: http://wiki.openstreetmap.org/wiki/Osmosis/DetailedUsage#--read-change-interval_.28--rci.29

PostGIS[edit]

PostgreSQL has been setup to store its data on the /sql partition:

ravar@cassini:/sql$ sudo ack -a /sql /etc/postgresql
/etc/postgresql/8.3/main/postgresql.conf
38:data_directory = '/sql/postgresql/8.3/main'          # use data in another directory

And for allowing password auth from localhost:

ravar@cassini:~$ sudo tail -n9 /etc/postgresql/8.3/main/pg_hba.conf
# "local" is for Unix domain socket connections only
#local   all         all                               ident sameuser
# IPv4 local connections:
#host    all         all         127.0.0.1/32          md5
# IPv6 local connections:
#host    all         all         ::1/128               md5

local   all         all                               trust
host    all         all         127.0.0.1/32          trust

Imported databases[edit]

At first I wasn't using --slim, which is less usable, see: http://lists.openstreetmap.org/pipermail/talk/2009-July/038786.html

TODO: Re-import with intarray support to support OSC updating: see [1]

With i18n support[edit]
createdb -E UTF8 -O gis gis
createlang plpgsql gis
psql -d gis -f /usr/share/postgresql-8.3-postgis/lwpostgis.sql
echo "ALTER TABLE geometry_columns OWNER TO gis; ALTER TABLE spatial_ref_sys OWNER TO gis;"  | psql -d gis

Create language list & style:

perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl > /sql/misc-data/languages.yml
perl /usr/local/src/osm-tools/osm2pgsql-style/create-wikimedia-style.pl --style  /usr/local/src/osm/applications/utils/export/osm2pgsql/default.style --languages /sql/misc-data/languages.yml > /sql/planet.osm/wikimedia.style

Import the planet:

osm2pgsql --slim -d gis -U avar -W -S ./wikimedia.style planet-090715.osm

world_boundaries[edit]

As documented on the OSM wiki:

avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/world_boundaries-spherical.tgz
avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/processed_p.zip
avar@cassini:/sql/world_boundaries/packed$ wget http://tileserv.openstreetmap.org/shoreline_300.tar.bz2
avar@cassini:/sql/world_boundaries/unpacked$ unzip ../packed/processed_p.zip 
Archive:  ../packed/processed_p.zip
  inflating: coastlines/processed_p.dbf  
  inflating: coastlines/processed_p.index  
  inflating: coastlines/processed_p.shp  
  inflating: coastlines/processed_p.shx  
avar@cassini:/sql/world_boundaries/unpacked$ tar xjvf ../packed/shoreline_300.tar.bz2 
shoreline_300.dbf
shoreline_300.index
shoreline_300.shp
shoreline_300.shx
avar@cassini:/sql/world_boundaries/unpacked$ tar xzvf ../packed/world_boundaries-spherical.tgz 
world_boundaries/
world_boundaries/places.shx
world_boundaries/world_boundaries_m.index
world_boundaries/world_bnd_m.shx
world_boundaries/builtup_area.shx
world_boundaries/world_bnd_m.dbf
world_boundaries/builtup_area.prj
world_boundaries/places.shp
world_boundaries/world_boundaries_m.shx
world_boundaries/world_boundaries_m.shp
world_boundaries/places.dbf
world_boundaries/places.prj
world_boundaries/builtup_area.dbf
world_boundaries/world_bnd_m.shp
world_boundaries/world_bnd_m.prj
world_boundaries/world_boundaries_m.dbf
world_boundaries/builtup_area.shp
world_boundaries/world_boundaries_m.prj
world_boundaries/world_bnd_m.index
world_boundaries/builtup_area.index
avar@cassini:/sql/world_boundaries/unpacked$ mv coastlines/* world_boundaries/* .
avar@cassini:/sql/world_boundaries/unpacked$ rmdir coastlines world_boundaries
avar@cassini:/sql/world_boundaries/unpacked$ ls -l
total 534M
-rw-r--r-- 1 avar users 3.4M 2007-03-10 10:05 builtup_area.dbf
-rw-r--r-- 1 avar users 273K 2007-03-10 10:05 builtup_area.index
-rw-r--r-- 1 avar users  513 2007-03-10 10:05 builtup_area.prj
-rw-r--r-- 1 avar users  13M 2007-03-10 10:05 builtup_area.shp
-rw-r--r-- 1 avar users 285K 2007-03-10 10:05 builtup_area.shx
-rw-r--r-- 1 avar users  16K 2007-03-10 10:07 places.dbf
-rw-r--r-- 1 avar users  333 2007-03-10 10:07 places.prj
-rw-r--r-- 1 avar users 7.0K 2007-03-10 10:07 places.shp
-rw-r--r-- 1 avar users 2.1K 2007-03-10 10:07 places.shx
-rw-r--r-- 1 avar users 4.3M 2008-08-10 20:48 processed_p.dbf
-rw-r--r-- 1 avar users 2.0M 2008-08-10 21:59 processed_p.index
-rw-r--r-- 1 avar users 357M 2008-08-10 20:48 processed_p.shp
-rw-r--r-- 1 avar users 3.4M 2008-08-10 20:48 processed_p.shx
-rw-r--r-- 1 avar users 3.0M 2009-07-10 04:01 shoreline_300.dbf
-rw-r--r-- 1 avar users 1.3M 2009-07-10 04:01 shoreline_300.index
-rw-r--r-- 1 avar users  86M 2009-07-10 04:01 shoreline_300.shp
-rw-r--r-- 1 avar users 2.0M 2009-07-10 04:01 shoreline_300.shx
-rw-r--r-- 1 avar users 4.0M 2007-03-09 11:31 world_bnd_m.dbf
-rw-r--r-- 1 avar users 246K 2007-09-09 19:28 world_bnd_m.index
-rw-r--r-- 1 avar users  355 2007-03-09 11:31 world_bnd_m.prj
-rw-r--r-- 1 avar users  47M 2007-03-09 11:32 world_bnd_m.shp
-rw-r--r-- 1 avar users 244K 2007-03-09 11:32 world_bnd_m.shx
-rw-r--r-- 1 avar users 659K 2008-03-30 21:42 world_boundaries_m.dbf
-rw-r--r-- 1 avar users  85K 2008-03-30 21:48 world_boundaries_m.index
-rw-r--r-- 1 avar users  355 2008-03-30 21:51 world_boundaries_m.prj
-rw-r--r-- 1 avar users 6.4M 2008-03-30 21:42 world_boundaries_m.shp
-rw-r--r-- 1 avar users  30K 2008-03-30 21:42 world_boundaries_m.shx

/var/www/tile-browse[edit]

Create a HTML page to browse each of our 279 languages:

avar@cassini:/var/www/tile-browse$ for lang in $(perl /usr/local/src/osm-tools/osm2pgsql-style/wikipedia-language-codes.pl | perl -MYAML::Syck -E 'say join q[ ], sort keys %{ LoadFile(*STDIN) }'); do cp .browse-template.html browse-$lang.html && lang=$lang perl -pi -e 's[(?<=osm-like/)(\w+)][$ENV{lang}]' browse-$lang.html; done

Category:OpenStreetMap --Cseki 04:22, 19 November 2009 (UTC)