Requests for comment/Localisation format

This is an RfC on a JSON based localisation format and structure for MediaWiki core and extensions.

Rationale
The current MediaWiki localisation files are based on PHP arrays. The files are executable, and this is a potential security risk. Aside from this, having PHP arrays, makes the localisation files not directly usable in JavaScript, which will be a follow-up stage to this change. Having JSON as the MediaWiki file format, will also make it easier to integrate modularly developed functionality into MediaWiki, while preserving all the MediaWiki internationalisation features.

Out of scope

 * Representation in resource loader for i18n files. A separate RFC will be submitted for this.

Requirements
Requirements are at least the following:
 * Have the messages in a non-executable UTF-8 file format.
 * The scope is only limited to  for now.
 * Core and extensions should be able to have multiple groups of messages, located in standard structure (for example, ).
 * All current MediaWiki i18n features must keep working.
 * The current extension localisation format should keep working for backward compatibility purposes, and converted extensions would get backward compatibility code that looks something like this:
 * The new format will be mandatory for all Wikimedia deployed code.
 * Conversion scripts should be made available, and be done for all Wikimedia-deployed extensions.

Proposal
The proposal is as follows
 * We will use UTF-8 JSON files using the same syntax to that used by jQuery.i18n, https://github.com/wikimedia/jquery.i18n. This provides complete support for  and basic authoring information, but does not support all the edges of the MessagesEn.php files, which will remain (replacing them is out-of-scope for this RfC).
 * The JSON translation files will be stored with actual human-readable Unicode characters and not Unicode character numbers. If this is a problem, it should be a job of for example ResourceLoader to resolve this.
 * Extension messages files will be expected in general to be in a directory called  in the root of the extension's repo, split by language (,  ,  , etc.), with support for multiple sub-directories for splitting up into keyed groups, auto-loaded.
 * Groups' keys are locally-scoped, allowing only, and auto-generated from the name of the sub-directory (if appropriate); the primary group will be automatically named the same as the extension. If you want to refer to default group distinctly from the extension's whole set of modules, refer to it outright (named other than  ).
 * We will introduce a new configuration variable,,  which allows the directory/directories for the extension to be loaded.   will continue to work but will be ignored in   if the same key is specified in.
 * Implementation will be completed in advance of the release of MediaWiki 1.23, running in dual-support mode, with support for the old format being retired in MediaWiki 1.24. As MW 1.23 will be an LTS release, this will ensure that users of existing extensions will be able to use them unaltered for several years.
 * Write conversion script to convert from PHP arrays to JSON files, and run it on all Wikimedia-deployed extensions. Complex splits (e.g. for MediaWiki core and VisualEditor) will be done manually by the Language Engineering and VisualEditor teams.
 * maintenance/language/messages.inc will become obsolete.
 * maintenance/language/messageTypes.inc will be converted to translatewiki.net configuration.
 * $messages of languages/messages/MessagesXx.php core files will be converted into in languages/i18n/xx.json. Conversion of other properties maintaned in MessagesXx.php is out of scope of this RFC.
 * jquery.i18n JSON could support fuzzy tags, but it does not at the moment. In the future, this could be implemented using an array in the @metadata.

Examples
An extension with multiple groups of messages might choose to split its files up as follows:
 * This will be loaded as wikimediamessages
 * This will be loaded as wikimediamessages/cclicensetexts
 * This will be loaded as wikimediamessages/wikimediatemporarymessages
 * This will be loaded as wikimediamessages/cclicensetexts
 * This will be loaded as wikimediamessages/wikimediatemporarymessages
 * This will be loaded as wikimediamessages/wikimediatemporarymessages

The extension would load its values using. This is short for, and would load all message files in the above example automatically, including the namespacing.

A more complex use case is for extensions that use libraries, where moving the internationalisation files into the root of the extension would split the import and make things more complicated. A (slightly artificial) example might be:

For this example, the messages files would be loaded using:

Implementations
A spike version implementing this, alongside a shim that lets the  file still be used as a bridge to the JSON files, exists for the VisualEditor project (split three ways ahead of a repo split) and OOjs UI.