Global templates/Proposed specification

  Татары, узбеки и ненцы И весь украинский народ, И даже приволжские немцы К себе переводчиков ждут. И может быть в эту минуту Меня на турецкий язык Японец какой переводит И в самую душу проник. —Осип Мандельштам

This is a draft proposal for the functional requirements for global templates and modules.

A one-page description of the proposal can be found at User:Amire80/Global templates draft spec/TLDR.

'''This is not a project that is being executed or planned to be executed by anyone at any defined point in time, at least not yet. This is just an idea, albeit a very detailed one.'''

The eventual goal is to form a cross-team and cross-project commitment on implementing these things, with proper architecture, product and project management, community engagement, etc.

This document does not try to go into the details of technical implementation in terms of storage, caching, delivery, PHP code design, etc. It only tries to define the requirements of how this feature will work from the point of view of the users:


 * 1) People who create and maintain templates and modules.
 * 2) People who create and edit pages that transclude templates and modules. This is includes all editors and all kinds of pages:
 * 3) * All levels of experience: From those who are completely new to those who have made thousands of edits
 * 4) * All kinds of editing tools: wiki syntax editing, Visual Editor, Content Translation, and others (even bot operators)
 * 5) * All wikis: Wikipedia, Wiktionary, Wikivoyage, Wikidata, Incubator, etc., and any new future projects
 * 6) * All languages: English, French, Russian, Spanish, Armenian, Persian, Zulu, Manipuri, etc.
 * 7) * All kinds of pages: Wikipedia articles, articles talk pages, user talk pages, community discussion pages, WikiProject pages, categories, template documentation pages, etc.

Elevator pitch
A large part of the functionality of Wikimedia sites is implemented in templates and Lua modules. In their current form they cannot be shared across different sites and languages and they are hard to integrate with modern ways of creating and editing articles—Visual Editor and Content Translation. This causes waste of contributors’ effort, and difficulties for new editors and smaller projects. Making it possible to share them across sites will make software development faster and more robust and will let editors focus more on writing text.

The Problem
General comment: Unless noted otherwise, all references to “templates” also apply to Scribunto Lua modules.

Templates implement features of Wikimedia sites. Some of these features are highly prominent, especially the infoboxes, the references, “citation needed”, and many others. All the readers see them, and all the editors run into them in almost every edit action. In addition, they implement many of the sites’ internal community management features: requests for deletion, requests for unblocking, expressing support in discussions, sorting articles for WikiProjects, etc.

Templates provide an efficient mechanism to quickly design, deploy, and use repetitive pieces of text and markup in many pages. However, templates also have several acute usability problems for all kinds of editors.

Wiki syntax editors
Templates are often difficult to understand for editors in wiki syntax. People who are experienced with using a particular template will probably identify it and be able to edit a page that includes it, but editors who are not familiar with it will have to look up its documentation when they encounter it, even if they are generally experienced otherwise. And editors who have low experience will be baffled by the cryptic text with curly braces, pipe characters, equal signs, etc.

Using a feature that is implemented as a template requires knowing the template’s name and typing it in curly brackets ( – ) or copying it from another page. This is not obvious to new users, and veteran users have to learn each new template separately, too.

Some wikis have gadgets that add buttons that insert templates that are common in that project to the editing toolbar. These are different in each wiki, even though a lot of the templates have similar functionality across projects and languages.

Visual Editor users
VisualEditor users have some advantages with using templates, however there are many issues with them there as well. In particular, there is a similar discoverability issue: In Visual Editor, all of the functionality of the templates is hidden behind the “Insert → Template” menu item, and the user has to know the template name before using it.

Visual Editor’s “Insert” menu has items for Math formulas, Egyptian Hieroglyphs, Musical scores, and some other functions that are implemented as extensions, but it doesn’t have items such as “Infobox”, “Citation needed”, “Unit conversion”, “Quotation”, etc. All the templates are the same kind of generic item.

There is one notable exception: Some wikis have the “Cite” button, which inserts footnotes with reference templates. However, it is an exception that proves the rule. It requires manual configuration for even basic functionality, this configuration is separate in each wiki, and as a consequence many wikis don’t have this button at all. Another comparable exception, which is being added in September 2019, is special support for “Citation needed” templates, but this also needs some custom configuration on each and every wiki to actually work.

Difficulties for editors who write in more than one project
Many templates exist in one project, but not in others, and often a template is available, but in a different form. Because of this it’s difficult or impossible to reuse skills that were acquired in one project: the functionality that the template provides is sometimes unavailable and sometimes it works differently. Note that this applies not just to wikis in different languages, but also to different wikis in the same language, for example English Wikipedia and English Wikisource.

For people editing in different languages, templates make translation harder. When translating a page, templates are much harder to handle than the article text (“prose”), whether the translation is done manually or with Content Translation. Users often have to skip the template or to correct it after the article was published. This also causes abandoning translations in progress, because template translation looks intimidating.

The most commonly reported issues in Content Translation are about templates.

Content Translation has a template adaptation feature, which automates some parts of this process, but it works only if a corresponding template exists in both languages, and if all the parameters were meticulously mapped by the template maintainers. This must be done for each template in each language separately and manually, and continuously maintained when the source template changes. This happens even though the templates’ function across the languages is the same most of the time.

Ideally, the templates and their parameters should be transferred to the translated page almost completely automatically, so that the translators could focus on writing the prose, as writing the prose is the area where human work is most needed.

A template can be exported from one wiki to another, but once this is done, the template becomes a forked copy. It either stays in the state in which it was exported, or continues to be developed separately, causing incompatibility. Sometimes people keep maintaining the different copies, but this is not robust and cannot scale for the hundreds of wikis that we have.

Template parameters can have the same functionality, but different names in different wikis. They can be adapted using TemplateData aliases, but this is a suboptimal hack: it is not what TemplateData was originally made for, and it has to be done manually for each language pair.

Templates mix together algorithmic logic, human-readable text strings, and formatting. Because of this there is no robust way to translate the templates’ user interface strings, as it is done with core and extensions.

Difficulties for editors in smaller wikis
A new wiki project is created by installing core MediaWiki and enabling a default set of extensions. In practice this doesn’t create a level playing field because a lot of the key features of larger wikis is implemented in templates: infoboxes, citations, maintenance hatnotes (such as ), etc.

The proposed solution: Summary
There are already many features of MediaWiki that are global across wikis: images (using Commons), blocking, user accounts (CentralAuth), preferences, user pages, user JS and CSS pages, and some others.

It must be possible to store templates and modules in a global repository, too, and to localize them as robustly as it is done with extensions.

Global templates and modules will empower template maintainers in all wikis by letting them collaborate more easily on developing the templates’ code.

Global templates and modules will empower translators and localizers by allowing them focus on translating just the user interface strings (“messages”), without having to look for strings in the code, and by allowing them to use the same skills and tools for translating templates and MediaWiki extensions.

Global templates and modules will empower content editors in all wikis to write and translate content that uses these templates without having to dive into the differences and to relearn different rules and skills in each wiki.

The syntax for developing templates and modules, and the general template maintenance and deployment cycle will not change, so all the skills that the template maintainers have acquired over the years will remain relevant.

All wikis will be able to use the global templates, but not forced to do this. Communities will keep their capability to override any global functionality, design, workflows, and data.

Localizing templates will be as convenient as localizing MediaWiki extensions.

Templates must be able to be semantic and global
Semantic means that other software components, especially Visual Editor and Content Translation, must have a general way to understand that a template exists and that it provides certain functionality, so that it will be possible to insert it into a page as an infobox, a citation, a maintenance tag, etc., and not only as a generic template. Currently, the closest thing there is to making templates semantic is TemplateData, but it only describes the template’s parameters. It doesn’t, for example, help Visual Editor add an “Insert infobox” button to the toolbar.

Global means that a template’s code must be maintained in one place and usable in all wikis.

Making templates semantic
Templates have never been robustly semantic, in the sense of being easy to handle by software that processes pages.

There are few examples of templates that were made semantic:


 * Various reference templates, which are usable from the Visual Editor toolbar “Cite” button. They require writing a lot of separate code to configure Citoid on each wiki that wants to use them.
 * “Citation needed”, which is being adapted to Visual Editor as of September 2019. It also requires configuration on each wiki.
 * Templates for mentioning users in the Flow extension, which require local configuration, too.
 * Some dump processing and research tools can parse English Wikipedia’s WikiProject assessment templates, which are usually added to talk pages.
 * The PageTriage extension is configured to work with English Wikipedia’s hatnote templates (also known as “tags”).

In the case of PageTriage, the extension essentially hard-codes a single wiki’s templates, making it unusable in other wikis without a significant rewrite. Even if the on-wiki configuration step is small, as it is with the Flow example, it still needs to be done. This doesn’t scale well for the 900 wikis that Wikimedia has, and the thousands that it will have in the future.

These things should be global by default, so that they will be immediately usable in at least a basic default configuration on all wikis at once by extensions, bots, dump analyzers, etc.

Storage and delivery
The global templates and modules can be stored in a central wiki (Meta, Commons, or a whole new wiki), and it can even be Gerrit or another repository.

The best solution is probably creating a new wiki that will store them, without getting mixed up with images, general community discussion, etc.

Using Gerrit as storage for templates and modules code is technically possible, but it would lose an important element of accessibility for template maintainers: editing a template in a wiki page is much easier and familiar to the vast majority of template maintainers than making Git commits and waiting for code review. Therefore, Gerrit probably shouldn’t become a way for storing the template and module code, at least not the primary one.

Global templates and modules must be stored in a common repository that can be edited by most wiki editors. Rules about blocking and special permissions should initially be similar to the rules in other wikis: everything should be open by default, and it must be possible to protect very common, sensitive, or frequently-vandalized templates. Further rules about protection levels can be developed by the editors community later.

How the templates are delivered to the target wikis is a question of internal engineering and architecture, as long as the other requirements are addressed. These questions were discussed in the past by some platform developers, for example around the Shadow namespaces project. This document tries to address related questions of how it works for the user who edits a page that uses a template, or who maintains the template itself—how to write it in a localizable way; how is it translated; how is it locally customized; etc. These questions weren’t addressed sufficiently in the previous architectural discussions on the topic.

There should be on-wiki tools for showing at least basic analysis of templates’ and modules’ usage on pages: the number of transclusions and invocations grouped by wiki, and lists of pages that use the templates and the modules. The feature that shows which templates does a page transclude while it’s being edited must continue working with global templates.

Templates must remain easy to modify
An important feature of how templates currently work is that they are edited just like wiki pages and immediately become functional after publishing without review or deployment. This is somewhat dangerous because a bad edit can ruin many pages, but the fact is that it works mostly well.

This ease must be preserved. Community members who maintain templates will most likely reject moving to a new system that will require them to learn completely new skills and drag every change through an exhausting review and deployment phase. This probably rules out storing the templates in Gerrit, unless, perhaps, the process for review and deployment will be much easier than it is for extensions.

It must be possible to make some templates non-global
Not all templates should be forced to be global.

In fact, some templates should be local because they implement a functionality that is unique to a certain language. By their nature, such templates don’t need to be translated, and there should be a way to give a hint to both human editors and translation tools (such as Content Translation) that they don’t need to be adapted, and can be skipped or substituted.

It must be possible to override some functionality or appearance of a global template
No community should feel that a functionality is imposed on it by some powerful external player, like the English Wikipedia community, the Wikidata community, the WMF, or anybody else. Global templates should be developed and used collaboratively for the common benefit. Most of the time it should work for everybody.

Sometimes some communities will have strong opinions about wanting to have particular functionality or design that will be different in their language or project, or to show an infobox with information that is different from what is shown in other projects, or not to show it at all. The capability to override things locally must be allowed from the start. (Or rather, it must not be taken away.)

A global template must be immediately usable in every wiki
Just like a global user page is immediately available in every wiki in which there is no local user page, a template or a module that is created on the global infrastructure must be immediately usable in every wiki. This must not require copying wiki pages, creating wrapper templates with a local name, administrator intervention, waiting for hours for caches to refresh, etc.

It must be possible to translate all user-facing strings
The user interface strings (also known as messages) of core MediaWiki, extensions, and some external tools such as Pageviews are translated conveniently and robustly on translatewiki.net. This localization process is familiar to at least some editors in all languages.

It’s currently not possible to do the same with templates. Multilingual sites such as Commons and mediawiki.org have the “TNT” system for translating some templates, but it’s very complicated and cannot be reused for Wikipedia, Wikisource, etc.

Ideally, it should be possible to translate templates just like core and extensions, using a wiki with the Translate extension.

The translated string must become usable immediately after the translation is submitted using the translate interface.

It can be possible to edit the user interface strings in raw wiki pages, but ideally they should be edited primarily through a dedicated translation interface.

The template documentation pages must be translatable as well. It's mostly enough to do it using the page translation feature of the Translate extension, but it may require some adaptations.

The language in which the strings are shown to the user
Templates are primarily used when they are integrated into content, so by default the translated messages must be shown in the wiki’s content language.

Some templates, however, are used as UI elements, so perhaps it makes sense to also allow showing the translated strings in the user language, when it’s different from the wiki content language. This may be particularly relevant for multilingual sites such as Commons, Wikidata, Meta, and mediawiki.org.

MediaWiki’s usual fallback chains must be used when a translation is not available.

Message keys and message storage
Messages should be represented as keys, similarly to how it is done in MediaWiki core, extensions, and tools.

Writing translatable strings will probably be the largest change in the template development process that template maintainers will have to get used to. Hard-coded strings will have to be separated and moved to messages organized by key. It must be made as easy as possible not only for the translators, but also for the template maintainers. Otherwise, they won’t actually do it, and the feature will be effectively rejected.

To easily make keys globally unique, it’s probably OK to automatically include the global template name in the message key.

Organizing messages
The Translate extension organizes messages by groups, also known as projects, which can be further organized by aggregate groups. For example, Article Placeholder, Score, and Poem are all groups that represent the corresponding MediaWiki extensions, and they are all included in the “Extensions used by Wikimedia - Advanced” aggregate group, along with many other extensions.

Projects that represent MediaWiki extensions are configured in YAML files in the translatewiki repository and shown in the Translate user interface in a project selector, also known as message group selector.

Since there are many more templates than extensions, some modifications may be needed in the way the Translate extension handles message groups to accommodate templates translation.

Each template should be a message group. Closely related templates should be grouped in an aggregate message group. They can be similar to the categories in which they are stored, and in fact, the categories may even be reused. Editing files in a Git repository to organize these message groups is probably not desirable, because it’s too complex and slow.

It would be nice to show group and template names as localized in the selector, but it’s also OK if they are shown in English. If it’s good enough for extension localizers, it’s good enough for template localizers, too.

Finding how to translate a template
Every template description page needs to have a direct link to translating it to the user’s language.

Some templates use Wikidata labels as part of their UI instead of hard-coding strings. This is done at the moment in Wikidata Infobox on Commons, Infotaula persona (Infobox person) in the Catalan Wikipedia, and in several other templates. These labels and values can be localized in Wikidata itself. Such usage cannot cover all the needs of template localization, but it is legitimate and useful for particular purposes. As long as this is properly described in the template documentation, this can continue to be used, and probably doesn't need special infrastructure adaptations. (Perhaps the translation of the relevant labels and values can be somehow integrated into the Translate interface for localizing the template, but this is optional.)

Considerations for modules
Lua modules can load and parse translatable MediaWiki strings, but there is no defined way for storing these strings for Lua modules that are maintained as wiki pages. It is possible to package Lua modules as parts of extensions, and then they are able to load translatable strings from the extensions’ i18/*.json files, but this is done in very few extensions at the moment. Rewriting templates in Lua may be a more robust solution from the engineering point of view, but Lua will not necessarily be embraced by all existing template maintainers, and their cooperation will be crucial to the project’s success, so this cannot be done to all templates.

Localizing the template name
The template may have a different name in every language, but it must be directly connected to the central storage

Global templates and modules are supposed to be immediately usable in all wikis without any extra steps, so it must be possible to transclude a global template in a local wiki page using its global name. The cross-wiki editors community shall decide what will be the policy for these global names.

Similarly to parameter names, templates may have different names in different languages, and this must be preserved. There must be a structured way to translate template names. Perhaps Wikibase sitelinks can play a role, but not necessarily.

If this is not done, editors will either avoid global templates, or wrap the global template in a local template with a translated name, and this will probably cause the template to lose the connection to the global entity. This is not desirable and misses the whole point of the project.

Template names must only be translated to languages that can be content languages of wikis. Translation to Formal German or British English are probably unnecessary. There may be a way to have aliases or redirects. Language variants, for example for Serbian and Chinese, must be supported according to these languages’ needs.

If a local template exists in a wiki and it has the same name as a localized name of a global template, the local template will be used. This is similar to how local files with identical names override global files on Commons, and how local messages in the MediaWiki space override the localization coming from the code.

Lua module names are often localized, too. Their names can be localized for direct invoking from wiki pages, but since code usually uses English-like identifiers, the internal global names should probably be preferred for use in the code itself, for example in  statements.

Localizing named parameters
Parameter names are different in every language. They are usually based on words in each language, so it’s important for editing the transclusion in wiki syntax conveniently.

Ideally, the global template should have generic internal parameter names that have translations to different languages. This is somewhat similar to Wikidata property name labels, but it can be simpler: since English is a kind of a lingua franca for software developers and templates are a kind of software, it’s probably OK to have English as the default source language rather than generic numbers as it is in Wikidata.

These translations of parameter names must be validated:


 * they must not include invalid characters
 * they must not be repeated within one template in one language
 * Anything else?

Automatic parameter translation
If the localized template names and all the localized parameter names are stored centrally, it will possible to have a simple service that gets a valid template with parameters, a source language name, and a target language name, and outputs a localized template. For example:

Input:

Output:

In visual editing and in 2017-style wiki syntax editing, simply copying and pasting a template from wiki in another language will do the parameter translation automatically.

For plain wiki syntax editing, there should be a simple way to operate this service, for example a special page or a dialog box where an editor can paste the template and the source language, and get the template with translated parameters.

In both cases only the names of the template and the parameters will be translated. Translating the parameter values is discussed separately.

Nameless parameters
Nameless numbered parameters must continue working, of course.

A decision is needed about how will their names be localized.

Translating parameter values
In addition to making the templates’ functionality and design shared, some thought must be given to making the template parameter values shared, as well as not shared.

Some parameter values are the same in all languages by their nature. For example an IPA pronunciation of a place’s native name (e.g. [dɛn ˈɦaːx] for The Hague), the year of foundation of a city, the chemical formula of a compound, etc. At least some of these should probably be stored in Wikidata and easily loadable in a template.

Some parameter values have to be translated or transliterated, for example people’s names, translations of country mottos, etc.

Global templates must make this possible, but in practice, these things are still often copied across wikis, and this must be taken into account as well.

Some parameter values can be reliably and predictably converted automatically, and the global templates infrastructure must support this. For example, number formats and digit characters are often different in Burmese, in languages of India, and in some other languages, but they can be reliably converted using simple software.

Text direction
Templates must adapt themselves to the text direction (ltr / rtl) of the wiki in which they are displayed.

It must be convenient to write a template in a direction-neutral way, with as little explicit right and left alignment as possible.

For easier translation of content, templates that are common in the largest projects must become global
The most popular source language for translation in Content Translation is English, by far. After it come Spanish, Russian, French, German, Catalan, Ukrainian, Italian, Chinese, and Portuguese. Because of this, it makes sense that the common templates in the most common languages, especially those in English, are the ones that are the most important to make global for the benefit of all other languages.

Somewhat paradoxically, however, the editors in these largest languages are also the least interested in making them global:


 * The templates already work well for them and most people there don’t directly care about the convenience of translation to other languages.
 * Rewriting the templates so that the strings will be translatable may be time-consuming and may force them to change their template maintenance skills.
 * Making the templates suddenly used by many more projects may make it harder to achieve consensus about making future changes in how the templates work.

This is more a consideration of practicality and community relations than a consideration of engineering, but it must be taken into account when making technical architectural decisions. Without doing proper preparation in this area, the whole project may fail.

As long as there are important common templates that are not global, Content Translation will have to keep supporting them. If the infrastructure for global templates is created, and migration of existing templates proceeds in a good pace, then Content Translation developers may consider stopping developing and some day deprecating the code for non-global template adaptation.

The pace of migrating templates from large wikis to the central repository can be one of the success metrics for the project.

It must be possible to use templates completely in both wiki syntax and in visual editing
It’s obvious, but should be mentioned anyway: Wiki syntax editing is not going away soon, and it must be possible to keep editing template transclusions in pages as it is done now. This must not become more complicated.

However, Visual Editor is increasingly embraced by both experienced and new editors, so every feature of how templates work must work well in both Visual Editor and wiki syntax editing.

Extra templates features
There are some extra features of core and extensions around templates. All of them must continue working, and may need updating for the global templates age.

TemplateData

 * Template description must be translatable, just like other user interface strings.
 * Parameter names, aliases, and descriptions must be centrally translatable.
 * Wikitext format parameter (inline, block, custom) must keep working.

TemplateStyles

 * There must be a possibility to write Template Styles pages in the same central repository as templates. The central style must be loaded by default, and it must be possible to override it locally.

TemplateSandbox

 * Special:TemplateSandbox must keep working.
 * It must be possible to edit a template in the central repository and preview it in a page in the target wiki.

TemplateWizard

 * Uses a wiki’s standard search to find templates. The results are presented in a list that might need to be changed to make the global/local status visible.
 * TemplateWizard gets its information for templates from the TemplateData API, so as long as that keeps returning the same structure there shouldn’t be any issues, and i18n is already working.

Wikibase

 * Wikidata can be used to bring in some parameter values from a central repository to the wiki. This is used productively in several wikis, among them French, Hebrew, Basque, Catalan, Estonian, and some others, although the actual implementation may differ. This must continue working, of course. Unifying the way in which it is done across different wikis may become one of this project’s most significant impact areas.
 * It may also affect the project of editing template values from within wikis.

VisualEditor

 * VisualEditor obviously needs to be able to insert both global and local templates.
 * VisualEditor shows a link to the template description page in the template editing dialog. This link should point directly to the global template when it is used.

Making templates easily reusable on non-Wikimedia projects may be desirable, too
Even though it doesn’t directly benefit Wikimedia projects, it may make sense to consider making templates easily reusable not only across Wikimedia projects, but also on other MediaWiki sites. Doing this will probably require some more work, but it may contribute to better modularization, and this may eventually benefit Wikimedia projects, too.

This is comparable to the capability of direct embedding of images from Commons on non-Wikipedia websites.

Imagine a world
Imagine a world in which every single human being can freely share in the sum of all knowledge and it's actually a really easy thing to do because templates are global:

(Note: The "With global templates" column assumes that the infrastructure is deployed in all Wikimedia wikis, and that the most often used templates are moved to the central infrastructure.)

Status
As noted above, as of October 2019, this page is only an idea, and not a commitment to implement a project.

The Platform Evolution project (2018) indicated some intentions to have support for global templates in the future. The page Platform Evolution/Recommendations discusses ideas for updating content modularity, and says:
 * ... “boxes” are an ideal focus area for creating modularity. They represent self contained features and also an opportunity to enable equitable sharing of user features across projects and languages be establishing a cross-project service to share templates. This project will also force us to consider how to handle content layout and structure separately from composable pieces content.

The closely related page Platform Evolution/Goals lists this as one of the goals:
 * Increase equity and power of contribution tools. We want to support the contribution of more content types of content, including media, in more interactive ways and across all projects. This means making some existing tools - like templates - available for consistent reuse across all projects and languages. It also means improving translation tools to remove silos of content. Finally, we also want to make it easy for contributors to create new cross-project, localizable content tools.

Other than these goals, however, there is no detailed plan for how such a feature will work. This page is an attempt to propose such a plan and listen to feedback from editing communities.

Useful links
Some relevant pages that discuss similar topics:
 * Platform Evolution/Goals
 * Platform Evolution/Recommendations
 * Multilingual Templates and Modules - An attempt to implement a similar feature using bots
 * meta:Community Wishlist Survey 2015/Results - Central Global Repository for Templates, Lua modules, and Gadgets came in as #3 in the Community Wishlist vote. Listed as “In development - Parsing team”, but not actually done.
 * meta:Which templates should be global? - an informal list made by various editors