Extension:LocalisationUpdate/LUv2

Background
Wikimedia wikis use the LocalisationUpdate extension to get the updated localised messages from translatewiki.net. To accomplish this, the extension downloads the translation files for mediawiki and installed extensions, stores them to local cache and then processes them to check which messages that can be updated.

Problem
The described flow is not very efficient since not all the wikis require support for ~300 languages and the extension downloads complete files instead of only downloading the deltas. This process consumes quite some time. Plus the extension requires setting up cron jobs and other manual configuration which can be avoided.

Enter LUv2
LUv2, Localisation Update v2, aims to fix this with creation of a new service which will keep track of the updates and make them accessible via a push interface and a RESTful api.

Everytime translatewiki.net admins commit translations to trunk the server database will be updated. The server will make these updates available to the clients through a RESTful api. In order to get the updates the client will make a post request to the server with the project_name, language_code, a list of message_id->hash pairs. The server will compare these hashes and respond with a list of updated message_id->message pairs, if any. The client will update the translation messages and the hashes.

The server will store the translations in redis. The server will provide an api endpoint which, when called, will trigger the update process. This flow will be automated using the post-receive hook on the projects' remote repo. The script will download the new files to a temporary directory, parse and compare them with the old files and insert/update the messages in redis. This script will ideally support all the formats that are currently supported by the translate extension.

There are npm packages available for parsing php arrays, appleFFS, Gettext, java .properties, ini, json, python dictionaries and yaml. Support for other formats will be added later if need be.

The redis will primarily be used store the following data. component_id   - Project identifier lang           - language code hash           - Hash of the message id             - Message identifier message        - the message

Deliverables

 * The LUv2 service + update mechanism
 * New LocalisationUpdate extension