Wikimedia Maps/Tile generation report

From mediawiki.org

Introduction[edit]

The map infrastructure store vector-tiles in a Cassandra DB and can store data from zoom levels 0 to 15 where each zoom level has 2zoom tiles. Tilerator generates 22 tiles per second in average for each CPU available. Generating the whole planet can take more than a month for 24 CPU's.

Tile generation Benchmark per zoom level for 24 CPU's
Zoom Total tiles Estimated Time to finish
Hours Days
0 1 0.00 0.00
1 4 0.00 0.00
2 16 0.00 0.00
3 64 0.00 0.00
4 256 0.00 0.00
5 1024 0.00 0.00
6 4096 0.00 0.00
7 16384 0.01 0.00
8 65536 0.03 0.00
9 262144 0.14 0.01
10 1048576 0.55 0.02
11 4194304 2.21 0.09
12 16777216 8.83 0.37
13 67108864 35.31 1.47
14 268435456 141.22 5.88
15 1073741824 564.89 23.54
Total 1431655765 753.19 31.38

Debian upgrade to stretch[edit]

Because of the Debian upgrade to stretch, OSM data needed to be reloaded and all tiles needed to be generated from scratch. The tile generation started at 10/22/2018 and it's an ongoing process until now (last update was 12/19/2018). After some delay on tile generation we started keeping manual log through time regarding the tile generation health using a node tool that collect metrics generated by Tilerator, which can be seem in the following table:

Tile generation health monitoring
Metrics for tile generation (last update 12/19/2018)
processed tilegenok tilenodata processedRate missing okTilesRate totalTiles timestamp
329605850 18235862 311369988 23.02% 1102049915 5.53% 1431655765 1543273715182
329631323 18236329 311394994 23.02% 1102024442 5.53% 1431655765 1543273762771
329656677 18236516 311420161 23.03% 1101999088 5.53% 1431655765 1543273810907
329678926 18236752 311442174 23.03% 1101976839 5.53% 1431655765 1543273851202
329839665 18238151 311601514 23.04% 1101816100 5.53% 1431655765 1543274169998
330184897 18244086 311940811 23.06% 1101470868 5.53% 1431655765 1543274814486
330242080 18244458 311997622 23.07% 1101413685 5.52% 1431655765 1543274924304
357986666 18878807 339107859 25.01% 1073669099 5.27% 1431655765 1543327057836
358478689 18890753 339587936 25.04% 1073177076 5.27% 1431655765 1543327999390
358481072 18890909 339590163 25.04% 1073174693 5.27% 1431655765 1543328003357
359466359 18915835 340550524 25.11% 1072189406 5.26% 1431655765 1543329887495
359468695 18915835 340552860 25.11% 1072187070 5.26% 1431655765 1543329890136
359501841 18916801 340585040 25.11% 1072153924 5.26% 1431655765 1543329957355
360883733 18960382 341923351 25.21% 1070772032 5.25% 1431655765 1543332566057
361105061 18968522 342136539 25.22% 1070550704 5.25% 1431655765 1543332988692
369736755 19204832 350531923 25.83% 1061919010 5.19% 1431655765 1543349355705
369754943 19205421 350549522 25.83% 1061900822 5.19% 1431655765 1543349393845
371169554 19214191 351955363 25.93% 1060486211 5.18% 1431655765 1543352016372
404768668 19856922 384911746 28.27% 1026887097 4.91% 1431655765 1543414812441
445075588 21248972 423826616 31.09% 986580177 4.77% 1431655765 1543490292663
448271613 21337248 426934365 31.31% 983384152 4.76% 1431655765 1543496265671
448305075 21338033 426967042 31.31% 983350690 4.76% 1431655765 1543496330227
448361068 21339359 427021709 31.32% 983294697 4.76% 1431655765 1543496440993
448376008 21339996 427036012 31.32% 983279757 4.76% 1431655765 1543496464526
448376958 21339996 427036962 31.32% 983278807 4.76% 1431655765 1543496466399
448444000 21342452 427101548 31.32% 983211765 4.76% 1431655765 1543496592815
448454022 21342935 427111087 31.32% 983201743 4.76% 1431655765 1543496611603
448455063 21342935 427112128 31.32% 983200702 4.76% 1431655765 1543496613191
448456352 21342935 427113417 31.32% 983199413 4.76% 1431655765 1543496616785
448463340 21343345 427119995 31.32% 983192425 4.76% 1431655765 1543496632324
448488739 21344399 427144340 31.33% 983167026 4.76% 1431655765 1543496680542
448493339 21344827 427148512 31.33% 983162426 4.76% 1431655765 1543496686367
454149805 21521204 432628601 31.72% 977505960 4.74% 1431655765 1543507258034
454419148 21531201 432887947 31.74% 977236617 4.74% 1431655765 1543507751739
459882332 21665626 438216706 32.12% 971773433 4.71% 1431655765 1543517792995
495010058 22871478 472138580 34.58% 936645707 4.62% 1431655765 1543582997617
495015752 22871535 472144217 34.58% 936640013 4.62% 1431655765 1543583005146
500146062 23023677 477122385 34.93% 931509703 4.60% 1431655765 1543592543127
501753227 23090414 478662813 35.05% 929902538 4.60% 1431655765 1543595574348
643485413 27850894 615634519 44.95% 788170352 4.33% 1431655765 1543858114418
643557080 27851243 615705837 44.95% 788098685 4.33% 1431655765 1543858251078
643575347 27851292 615724055 44.95% 788080418 4.33% 1431655765 1543858283434
643588132 27851365 615736767 44.95% 788067633 4.33% 1431655765 1543858306499
647378317 27923230 619455087 45.22% 784277448 4.31% 1431655765 1543865399600
647390779 27923944 619466835 45.22% 784264986 4.31% 1431655765 1543865417667
554720434 23601894 531118540 38.75% 876935331 4.25% 1431655765 1543935910238
554724936 23601894 531123042 38.75% 876930829 4.25% 1431655765 1543935916982
571188609 24069424 547119185 39.90% 860467156 4.21% 1431655765 1544026116760
586166907 24103862 562063045 40.94% 845488858 4.11% 1431655765 1544053956011
568689760 22641417 546048343 39.72% 862966005 3.98% 1431655765 1544156936945
589400156 22071213 567328943 41.17% 842255609 3.74% 1431655765 1544221066989
589418129 22071231 567346898 41.17% 842237636 3.74% 1431655765 1544221094608
637406144 21566271 615839873 44.52% 794249621 3.38% 1431655765 1544394201838
637408830 21566273 615842557 44.52% 794246935 3.38% 1431655765 1544394205758
637411640 21566273 615845367 44.52% 794244125 3.38% 1431655765 1544394209308
680100749 22261714 657839035 47.50% 751555016 3.27% 1431655765 1544472254313
717843703 22810362 695033341 50.14% 713812062 3.18% 1431655765 1544541762797
718145412 22816663 695328749 50.16% 713510353 3.18% 1431655765 1544542312249
756847246 23204744 733642502 52.87% 674808519 3.07% 1431655765 1544641669818
773737425 23618129 750119296 54.04% 657918340 3.05% 1431655765 1544710360518
819218612 23969150 795249462 57.22% 612437153 2.93% 1431655765 1544794097109
820425720 23984577 796441143 57.31% 611230045 2.92% 1431655765 1544796310087
826459063 24036728 802422335 57.73% 605196702 2.91% 1431655765 1544807325635
863417905 24334279 839083626 60.31% 568237860 2.82% 1431655765 1544875191371
962442186 26073481 936368705 67.23% 469213579 2.71% 1431655765 1545057598335
1056702638 27869655 1028832983 73.81% 374953127 2.64% 1431655765 1545228533232
1057790348 27879600 1029910748 73.89% 373865417 2.64% 1431655765 1545230495655
1078537988 28093576 1050444412 75.34% 353117777 2.60% 1431655765 1545268190817
1078577797 28094882 1050482915 75.34% 353077968 2.60% 1431655765 1545268269267
1078605334 28095272 1050510062 75.34% 353050431 2.60% 1431655765 1545268313809
1078621315 28095300 1050526015 75.34% 353034450 2.60% 1431655765 1545268340157
1078662686 28095300 1050567386 75.34% 352993079 2.60% 1431655765 1545268411926
1078675899 28095300 1050580599 75.34% 352979866 2.60% 1431655765 1545268439748
1078681693 28095300 1050586393 75.35% 352974072 2.60% 1431655765 1545268451792
1114218777 28670702 1085548075 77.83% 317436988 2.57% 1431655765 1545332461909
1117927315 28776224 1089151091 78.09% 313728450 2.57% 1431655765 1545339308889
1146342225 29491813 1116850412 80.07% 285313540 2.57% 1431655765 1545391152361
1150046974 29624996 1120421978 80.33% 281608791 2.58% 1431655765 1545397953049
1150162021 29625299 1120536722 80.34% 281493744 2.58% 1431655765 1545398162508
1150170358 29625299 1120545059 80.34% 281485407 2.58% 1431655765 1545398177374
1212590800 30434496 1182156304 84.70% 219064965 2.51% 1431655765 1545512936226
1212608207 30434600 1182173607 84.70% 219047558 2.51% 1431655765 1545512969903
1714812975 37884031 1676928944 119.78% 0 2.21% 1431655765 1546432643215
1715522034 37884184 1677637850 119.83% 0 2.21% 1431655765 1546433933383
1957220075 38965688 1918254387 136.71% 0 1.99% 1431655765 1546869993652
1374582898 34267596 1340315302 96.01% 57072867 2.49% 1431655765 1546961480404
1377981578 34267596 1343713982 96.25% 53674187 2.49% 1431655765 1546967794888
1416730878 34267675 1382463203 98.96% 14924887 2.42% 1431655765 1547040742682
1419676228 34267736 1385408492 99.16% 11979537 2.41% 1431655765 1547046408039
1340948139 37151623 1303796516 93.66% 90707626 2.77% 1431655765 1547133254764
1342254006 38076370 1304177636 93.76% 89401759 2.84% 1431655765 1547138839505

It is possible to see that Tilerator processed more tiles than expected and also, after some jobs failing with message "TTL exceeded" the number of tiles processed dropped and Tilerator rerun the jobs. It's not known if the work is committed and tiles were stored in Cassandra, even though maps1004 increased its size of generated tiles compared with maps1001 (old master). The remaining jobs were stopped at 2019-09-01 15:23 freeing CPU to enable the OSM sync replication script.

General Updates[edit]

Tile processing reached near 100% (01-08-2019)[edit]

After a long run of tile generation jobs, and some difficulties to reach an end, the team decided that would be reasonable to continue with the migration plan without waiting for the full tile processing. Analyzing the collected data we figured out that only a small part of the tiles are stored and we might have enough tiles stored.

Also, it was found a failure in the strategy to regenerate the full planet tiles. It was documented a different process that is faster if you have a set of tiles pre-generated before to use as a basis to generate only the specific indexes. The next time that full planet tile regeneration is needed, we should consider this approach which would generate tiles according to the following benchmark:

Tile generation Benchmark per zoom level for 24 CPU's
Zoom Total tiles Estimated Time to finish
Hours Days
0 0 0.00 0.00
1 4 0.00 0.00
2 16 0.00 0.00
3 57 0.00 0.00
4 218 0.00 0.00
5 846 0.00 0.00
6 3301 0.00 0.00
7 13024 0.01 0.00
8 51509 0.03 0.00
9 202145 0.11 0.00
10 159340 0.08 0.00
11 526282 0.28 0.01
12 1822626 0.96 0.04
13 6099772 3.21 0.13
14 19000727 10.00 0.42
15 61208604 32.20 1.34
Total 36.87 1.54

Is important to notice that after the tile regeneration we had an increase of tiles on maps1004. For example, z15 increased the most with 61,208,604 tiles, that is 12,749,948 more tiles than our current production (~26% increase).

Useful Commands[edit]

The following Tilerator command was used to schedule and queue tile generation jobs:

sudo -u tileratorui node /srv/deployment/tilerator/deploy/node_modules/tilerator/scripts/tileshell.js --config /etc/tileratorui/config.yaml \
-j.zoom 0 \
-j.fromZoom 0 \
-j.beforeZoom 16 \
-j.generatorId gen \
-j.storageId v4 \
-j.deleteEmpty true \
-j.keepJob true

Problems during tile generation[edit]

  • Tile generation for zoom levels 14 and 15 failed. Tilerator tries to run jobs with too much tiles and got the error TTL exceeded.
    • Solution: restart jobs using Tilerator UI

What we have learned[edit]

A few things that was learned during this process are worth to mention:

  • Generating an entire zoom level is painful and slow and we should avoid at all costs for zoom levels greater than 13
  • This work is an incentive for a refactor of the available documentation
  • There isn't yet a reliable tool for Cassandra data copy, which would save us a lot of time too