Language tools

The Language team has the goal of reducing the language barrier that users face when accessing or sharing knowledge. The team has created tools to facilitate the participation of our users regardless of their language. These include the creation of multilingual content and software, facilitate language navigation, and support for easy input and proper display of different scripts. Language tools and their infrastructures need regular maintenance and update efforts. Making our language tools and infrastructure work better helps all the projects that rely on them, and saves time and efforts to the people participating in them, from developers to translators and the final users.

In July 2018, an effort was started to identify key areas to improve in our language tools during the upcoming months. The affected tools, the plan and the status updates are described next.

List of tools

Language tools are listed below:

• Translate extension . The infrastructure used to translate our software and multilingual pages. Communities of translators use it on Translatewiki.net and Wikimedia Meta-wiki.
• Universal Language selector . Provides easy access to language settings, and facilitates language selection in different contexts including interlanguage links .
• Project Milkshake. Developer libraries for internationalisation support on the web including input methods, web-fonts, and grammatical rules for languages that can be used on both Wikimedia and external projects.
• MediaWiki Language Extension Bundle . A collection of selected MediaWiki extensions needed by any wiki which desires to be multilingual.
• Content Translation . The tool to facilitate translation of Wikipedia articles across languages. Since it is in active development, the progress of this tool won't be tracked as part of the current maintenance effort, but as part of its own project page for the new version .

Maintenance plan

Since there is a wide catalog of language tools and potential areas to improve we plan to operate in the following way:

1. Language tools audit. Identify areas and their potential impact in order to have clarity on the areas we want to focus our maintenance work regarding language tools.
2. Get feedback. Share our plans and get input from the different people involved, including the users of the tools and people working in other Wikimedia projects.
3. Work on the selected areas. Once areas are identified and the people involved are in the same page, the Language team will work on the areas defined.

Your input is important in this process, so feel free to add your comments to the talk page.

Status updates

September 2021

• Supporting improvements to the language button.
• Preparing for translatewiki.net operating system upgrade to Debian 11. Making compatibility fixes to SMW.
• Working on to resolve two production issues with the Translate extension: one JavaScript error and one about inefficient database queries

August 2021

• Published blog post about recent support to send messages in the best language available.
• Improve exports by simplifying their processing to use a single invocation.

July 2021

• Automating key steps of the localization process by supporting a stronger synchronization in RepoNG and Translate, preventing all translation units to become outdated when moving a translatable page, and avoid log entry changes to consider a translatable page as broken.
• Provide access to language settings on new Vector from the language selector.
• Make the localization platform more reliable by avoiding messages to become empty when translate din page mode, and issues with the trim function and with message group stats.

June 2021

• Due to an outage during translatewiki.net deployment caused by the way we deploy LocalisationCache, this is now part of the deployment tag.
• We improved Gettext file format parsing code to avoid situations where source language strings would incorrectly be imported as translations.
• Released MLEB 2021.06.
• Translate is no longer compatible with MediaWiki 1.34.
• The Translate extension no longer uses deprecated methods, allowing them to be removed later once all other uses elsewhere have been removed.

May 2021

• Released language-data 2021.04.
• Once a MessageUpdateJob has failed, the administrator can now investigate the issue, and has the ability to mark the error with the message as resolved.
• Created a maintenance script for moving big translatable pages so that there is a reliable way to perform these large renames.
• Added Wikimedia OCR to translatewiki.net.
• Yearly statistics are now available in Special:TranslationStats.
• Corrected code that caused a silent and permanent deletion of settings of translatable pages when associated translation page was deleted so that now translatable page settings are no longer disappearing unexpectedly.
• Translatable pages consisting of too many pages can no longer be moved through the web interface; such pages can be only moved using a command line maintenance script.
• Created a "Getting started" guide for developers that covers the following topics: (1) Local development environment setup, (2) How to pick tasks, and (3) Coding conventions.
• It is now possible to create pages with "message bundle" content format that includes basic schema validation.

April 2021

• Until current errors can be resolved by the translate administrator, they will be tracked via the Group synchronization cache.
• Added Cita to translatewiki.net.
• Added a new JSON file containing partner descriptions to translatewiki.net.
• Clarified and updated Ojibwe language codes on ranslatewiki.net.
• Improved wrapping of page contents delivered using MassMessage to avoid regressions.

March 2021

• Added EDTF (PHP library) to translatewiki.net.
• Added Wikidata Lexeme Forms to translatewiki.net.
• Resolved fatal error that prevented exporting.
• Translatable templates are usable on translatable pages with the normal syntax for transclusion.
• Limited the max number of translatable pages that can be moved via Special:MovePage.
• To facilitate the Translate Extension’s transition to be compatible with the new Parsoid, we explored an approach that makes it easier for users to edit translatable pages, and could inform improvements in both Visual Editor and Parsoid.

February 2021

• Released MLEB 2021.01 on 28 January 2021.
• Translations are no longer incorrectly shown as outdated due to message definition getting out of sync.
• Definitions, translation and message documentation with no content are now treated in the same as those with content to avoid build/CI failures.
• There is a knob for a removal threshold, which is configurable per project and it defaults to 0% now, meaning files without translations are deleted.
• Previously, when a language fell under the export threshold (% of translations completed), it would no longer get any updates from translatewiki.net. This lead to problems and now we keep updating such languages.
• Message groups with only optional messages now get exported if other conditions are met.
• Met with Brian Choo, Patrick Earley, and Samuel Guebo regarding the Universal Code of Conduct's translation process so that the language team could better understand their translation situation and come up with solutions for next time to avoid errors and issues.
• Added WS Export to translatewiki.net.

January 2021

• A Retrospective was held on 2020-12-22 on translatewiki.net deployment practices; new tasks were filed as a result.
• It is now possible to access language selector for input methods on Special:Translate as previous issues were fixed.
• Added Toolhub to translatewiki.net as a new project.
• Added VideoCutTool to translatewiki.net.
• Added Miraheze Landing Page to Translatewiki.net.
• Reverted changes due to errors caused by incorrectly mapped code paths and tables.
• A bug that caused Special:SupportedLanguages to not work on translatewiki.net was fixed.
• It is now possible again to move translatable pages on translatewiki.net and in other wikis outside Wikimedia production.
• Translatewiki.net translation administrators can again use the message key rename support of Special:ManageMessageGroups instead of having to spend more time to perform those renames manually.
• We fixed a small style regression that caused inconsistent styling when using the Timeless skin caused by a change in the Timeless skin.
• The display of a button on Special:ManageMessageGroups was broken while updating, but this was resolved.

December 2020

• Updated code so that title prefixes are only displayed if there is something to display, reducing log errors.
• Removed FCFontFinder as it is no longer used in the Translate extension.
• Updated the Phabricator template to make adding a new project to Translatewiki.net easier.

• Restored the help link and its functionality on Special:AggregateGroups when using the Timeless theme.
• Corrected code used to fix a recent regression that broke the statistics display on the main page of translatewiki.net.
• Wikidata query builder was added to translatewiki.net, providing a visual interface for building a simple Wikidata query.
• Addressed PHP 7.4 notices associated with MediaWiki 1.35.0 in the next MLEB release.
• Reviewed and added more Etherpad lite plugins to translatewiki.net based on user feedback.

• Added Anvesha application to translatewiki.net as a user’s request.
• Adjusted which folder Insertables to are mapped to, improving user experience.  
• Added support for sending only the specified section of a page for both non-translatable and translatable pages.

November 2020

• MediaWiki Language Extension Bundle (MLEB) 2020.10 was released, see release notes here.
• New system admins are now able to do puppet changes for translatewiki.net after reading our new documentation.

October 2020

• Improved accessibility for the Translation statistics by enabling screen readers to access translation statistics graphs.
• Added in syntax validation step for yaml files in the translatewiki repository
• Enabled Wikipedia IOS to use the Pull Request (PR) feature on translatewiki by reviewing, approving, and merging appropriate PRs.
• Enabled Wikipedia Android to use the Pull Request (PR) feature on translatewiki by reviewing, approving, and merging appropriate PRs.
• Added official etherpad-lite plugins to translatewiki.net.
• After correcting sync errors caused by replication lags, all translation pages are now showing all the latest changes.
• Corrected a typeError that caused Webfonts to be on the incorrect page.
• Replaced phplot graphs with JavaScript library solution to improve user experience.
• It is not possible to produce statistics about registrations using Special:TranslationStats or translationstats Action Api, as a result we removed this metric.
• Corrected an MW_Version error associated with MLEB 2020.07.
• Released an updated version of Language-data.
• Improved code health by removing unnecessary code that supports MW 1.33 and below now that MW 1.35 has been released.
• Published documentation explaining the current translation memory architecture to improve user understanding.
• Added SWViewer to translatewiki.net in support of detecting and reverting vandalism.
• Unparsed <bdi> tags are no longer shown in the priority languages warning in Special:Translate.

September 2020

• Fixed the error that caused translation aids to not work for one message group in Wikimedia Commons.
• For translatable pages, added syntax to prevent wrapping in places where text wrapping is inappropriate.   
• A change in MediaWiki core caused all patches for Translate to fail code quality tests which effectively blocked merging all changes; the unavoidable test failures were suppressed.
• For translatable pages, added a wikitext magic word which returns the correct language for a translation unit depending on if it is translated or not.
• Translation page output is now tagged correctly with regards to language and writing direction.
• Translators are now able to translate pages recently marked for translation with a decrease in production issues related to the message index.
• Updated code so that any page with custom translation unit identifiers containing the letter n can now be marked for translation.
• Numbers are now formatted properly as the language allows in the translation statistics graph.
• Altered TranslationStats graphs so that they are oriented horizontally.
• Corrected errors that lead to the wrong tvar information to display.
• Converted Special:MovePage to OOUI.
• Improved documentation for what happens when a page is marked for Translation with diagrams and descriptions.

August 2020

• A new release of the language-data library is now available on npm and packagist, see changelog here.
• MediaWiki Language Extension Bundle 2020.07 was released, see release notes.
• Volunteer translators are now able to support translation of Wikipedia Preview elements.
• New developers are aware of the local conventions when they change Translate code, due to the new Developer Guide for the Translate extension.
• Project Dibabel was added to translatewiki.net to help synchronize template and module pages with the master copy.
• Fixed broken language selector on the Special:NotifyTranslator page that caused javascript errors.

July 2020

• Confirmed resolution of incorrect translations mapped to Catalan when they’re from non-catalan languages.
• Corrected errors that broke regular software translation updates from translatewiki.net, caused by Gerrit upgrade.
• Added support for sending multilingual messages using translatable pages, including fallback languages, to the Mass Message extension.
• Fixed parsing of pluralization markup in translations to avoid export errors on translatewiki.net.
• The WPCleaner project was added to translatewiki.net.
• It is now easier for editors editing translatable pages to find documentation describing the Translate extension specific markup.

• Author attributions in translation files were restored after being incorrectly transferred.
• Corrected errors that happened while editing messages documentation.
• New translatable pages will use the new syntax version, old pages can now be migrated by translation admins by ticking a checkbox when marking a page.
• Resolved errors that led to page freezes when using the translation interface and translation target language was equal to the source language.
• Reduced Technical debt in Translate, improving the experience for Translate extension users.
• Corrected errors that lead message groups to cause console errors and not appear on the search translation page.

June 2020

• Translation memory suggestions on Wikimedia projects are no longer duplicated and now suggestions are loaded faster.
• Translatewiki.net MediaWiki debug log files are now rotated, limiting the space they use and automatically purging potentially sensitive information after a given time period.
• Fixed the Translation Notification extension so that translation notifications are now reliable and translation admins can send them without restrictions.
• Released Language-data library in both Packagist and NPM package repositories.
• Added convenient-discussions, a JavaScript tool that allows the user to post and edit comments without switching to a separate page, to translatewiki.net.
• Improved handling of the backslash character in translations exported from translatewiki.net, reducing errors.

• Resolved errors on TranslatorSignup so that it is possible to select the wiki where you want a notification to be delivered.
• Added project Skin:Citizen, a fully responsive skin compatible with any MediaWiki installation 1.31 or higher, to translatewiki.net.
• Oppia project now receives translations through pull requests to avoid build failures in development branch.
• Translation notifications are now reliable and translation admins can send them without restrictions.

May 2020

• Released the most recent version of MLEB.
• Corrected the popup box for subprojects disappearing immediately on Translatewiki.
• Updated translate extensions to not reparse certain files, reducing errors.
• Removed incorrect count columns from Special:Statistics to reduce confusion caused by displaying out of date information.
• Enabled Kiwix Desktop 2.0 for use with Translatewiki.
• Enabled default fonts for translatewiki, correcting a past error.
• Resolved MediaWiki console errors, improving functionality and decreasing spam in error logs.
• Resolved errors that lead to failed Translation statistics reports, improving reporting.
• Corrected errors that caused translated subpages to not move correctly with the parent pages.
• Updated code to now support offline translation exports from the command line.
• Corrected the fatal error of an uncommon code path in the Translation extension due to a missing autoloader entry.
• Removed unnecessary language statistics and validation scripts from MediaWiki core since checks are performed on the translatewiki.net side.

April 2020

• Making the list of supported languages more reliable by solving fatal errors.
• Improved QA process by providing access to code changes before deploying them to all users ("canary")
• Better documentation and code quality for managing localization repositories.
• Better access to translation context by fixing the access to instructional documentation.
• Added localization support for Page Content Service.
• Increased the quantity and quality of support apps by adding Bitmark/SPRING support for mobile apps
• Increased translation visibility by adjusting the workflow surrounding translation exporting
• Improve code quality to reduce PHP warnings
• Better offline translation support by fixing issues in the export process
• Increased stability of some translate extensions by fixing errors
• Improved accuracy for the code that updates language name
• Improve code quality to fix errors associated with certain database transactions
• Fixed code errors that prevented adjusting the title and namespace when moving a page in the Chavacano de Zamboanga (cbk-zam) Wikipedia
• Refactored and updated code for Translatewiki.net to increase speed of queries and reduce the chances for outages.

March 2020

• Analyzed the performance issues with Central Notice localization messages, identifying the cost of dealing with a large number of smaller message groups and defining follow-up steps to improve the way those are handled.
• Migrated WikimediaTools projects to the new framework and added plural validation for the ajapaik-app project for more flexible validation of translated messages. Projects previously migrated to the new validator framewok present no issues after the observation period.
• Removed the deprecated checker code to complete the migration to the new validator framework after the observation period.
• Reorganization of insertables (quick suggestions for the user to add when translating) to reduce redundancy of frequent cases defined by each project.
• More reliable translations by applying the new validator framework to check variables in the ajapalk-app project.
• Localization support for QRMedia project added.
• Improved compatibility of Translate with recent PHP versions.
• Added localization support for the COVID-19 poster from the Finnish Institute for Health and welfare.
• Better support for message renames by fixing an error when fetching content of unavailable messages.

February 2020

• Cleaner codebase by removing backward compatibility code to support old MediaWiki versions.
• Better communication of language changes by asking for user confirmation when accessing a link that request a change of language.
• Migrated 9 projects to the new framework for more flexible validation of translated messages: Pywikibot, Citation Hunt, Internet Archive Bot, GapFinder, Wikipedia Library Card Platform, MantisBT, Mazi, Mifos, and FUDForm.
• Migrated Android Mobile and lib.reviews projects to the new framework for more flexible validation of translated messages.

January 2020

• Migration of Open Street Map (website) and FreeCol projects to the new validator framework.
• Improved support for validating plural forms in translation messages to ensure consistency with the source message and the plural rules of each language.
• Updated auto-size libraries and data formats used.
• Improved performance of translation export process to prevent timeouts.
• Cleaner configuration for MLEB releases.

December 2019

• Simplified access to similar messages.
• More reliable import process for translations by ensuring the information is immediately updated at the end of the process.
• Improved saving process by keeping the list of languages immediately up to date.
• Improved performance when selecting a message group to translate.
• Improved validation by allowing to enforce that translation keys use certain values, and limit the validation to specific messages.
• Migration of Nocc to the new validator framework.
• More reliable diffs for translation messages.
• Support for language variants in the title of multilingual pages which was not possible to convert before.
• Technical updates for translatewiki.net related to SSL and searching.
• Improved stability with improvements to performance when loading translators, processing fuzzy translations, and less noisy certificate-related notifications.

November 2019

• Improved import process for translations by better presenting and handling messages that were renamed.
• Better context for translators by providing access to other messages using the proposed machine translation suggestions.
• Better approval process for translators by correcting issue with duplicate approvals.
• Support localization of DiscordWikiBot
• Improved integration of language selection into the personal toolbar by adjusting the visual alignment.
• Better representation of HTML special characters in translation suggestions.
• Fixed issues with extension missing path information.
• Improved control on translations with an option to allow translations only for the defined priority languages.
• More reliable export process by preventing failures in one project to prevent exporting others.
• Improved backup system for translatewiki.net

October 2019

• Improved database access for Translate
• Support for Global CheckUser on translatewiki.net
• Translation support for WhoWroteThat extension.
• Support for changes in AbuseFilter and SpamBlacklist APIs
• Reduce visual noise by removing separator in translatable pages.
• Support for localization of KaiOS Wikipedia app and Femiwiki.
• Compatibility support for language tools with new PHP versions, with older MediaWiki version and with function deprecation of new MediaWiki versions.
• More clearly surface duplicated content errors before users mark pages for translation.

September 2019

• Export repo migration for Encyclopedia of Life project.
• Avoid running Hip Hop Virtual Machine (HHVM) specific tests when such virtual machine is not used.
• Deprecation alerts as part of the migration process to use the new registration system for extensions.
• Improved support for moving translated categories and database load balancing.
• Improvements in the validation framework to check consistency of empty lines, and migration of a new project (Dissemin) to use the new validation framework.
• More reliable support for marking translations as discouraged.
• Migration of a new project (WMCZ-Tracker) to the new validator framework.

August 2019

• Improvements in the sign up process for Translate extension
• Fixed issues that made documentation for translations link to the wrong namespace, and issues from deprecated parsing functions.
• Better alignment with new standards by using the new extension registration system for Translate and for translatewiki.net, avoid using deprecated functions when moving translatable pages, and fixing issues with export functions on new PHP versions and the Hip Hop virtual machine (HHVM).

July 2019

• Improved translation message integrity by supporting custom checks with a new validation framework.
• More fluent account creation process by better communicating the wait to the user and making username available for previously rejected user applications.
• Better support for exporting by preventing issues with empty groups of messages to translate and support for Phabricator translations.

May 2019

• More reliable sharing/bookmarking of messages to translate by ensuring the link considers the current language when it changes.
• Better support for source language by preventing actions that do not apply such as proofreading, or changing the translation status.
• More clarity to communicate priority languages by showing a proper notice when translation to non-priority languages is blocked, avoid showing the message when already translating to a priority language, avoid showing the notice when translating documentation, and including also other languages in the stats.
• Better support for importing translation updates by clarifying the possible merge actions, and the link to import pre-existing translations.
• More reliable language selection by avoiding Simple English to be missing in the interlanguage links when it should be available.

April 2019

• Improved font support for Church Slavonic.

February 2019

• Better handling of searches with a large number of results.

January 2019

• More reliable update for translation statistics.

December 2018

• Support for Microsoft translation restored.
• Message group stats improvements to include review stats and avoid timeout errors.
• More reliably showing the progress of translations.

November 2018

• Improvements on code review infrastructure to prevent false positives in code review.
• Improvements in the list of message groups to provide more clarity by adjusting the colors.
• Better support for moving translatable sub-pages by avoiding moves to affect the parent page.
• Better proofreading support by avoiding translations to be automatically marked as reviewed when they are not.
• More reliable access to message groups by avoiding errors due to the truncation of long IDs, improved query performance, and avoided warnings.

October 2018

• More reliable support for moving and deleting translated pages.

September 2018

• List of proposed interventions documented on wiki for discussion.

July 2018

• Initial discussion on initial potential areas based on the Language goals and whishlist page.