Help:Content translation/Translating/Translation quality

When creating a translation, it is essential to review the contents before publishing them.

You need to make sure that the content produced is not altering the original meaning, and check that it reads natural in the destination language.

The initial machine translation provided helps to speed the translation process with a useful starting point, but the tool encourages users to review and edit significantly the initial contents.

Different mechanisms guarantee that translators edit the initial translations appropriately.

The translation editor tracks how much the initial translation is modified by the user, and defines different limits to either prevent publishing, or warn users to encourage them to further review the contents.

In this way, the tool makes it possible for users to use machine translation when they make a good use of it, while prevents the creation of lightly reviewed low quality translations.

More detail about how these limits work, how they can be adjusted to the needs of each language, and how to measure the quality of the content produced with the tool is provided below.

Limits to encourage reviewing the translation
Content translation [https://github.com/wikimedia/mediawiki-extensions-ContentTranslation/blob/master/doc/MTAbuseCalculation.md measures the percentage of modifications] that users make to the initial automatic translation provided.

In this way, the system knows how many words have been added, removed, or modified from the initial translation.

These measurements are made at two different levels: for each paragraph and for the whole translation.

Different limits are applied at each level, as detailed below.

Limits for the whole translation


You cannot publish a translation with 99% or more of unmodified contents for the whole document.

This limit is intended to block the most clear vandalism.

This prevents users from just adding paragraphs to the translation and publishing them with no edits at all.

As detailed below, this limit can be adjusted on a per language basis.

Limits for each paragraph


For each paragraph, the percentage of user modifications is also measured.

A paragraph is considered problematic when the paragraph contains more than 85% of the initial machine translation (or, when copying the contents from the source document, it contains more than 60% of unmodified content).

The translation editor will show a warning for each paragraph that is considered problematic, encouraging the user to edit it further.

In some cases users are still able to publish, but the resulting page may get added to a tracking category of potentially unreviewed translations for the community to review.

In other cases, users may not be allowed to publish.

These are some of the factors considered for determining whether to allow the user to publish or not (some of these are still in development):


 * The number of problematic paragraphs. Users are prevented from publishing translations with 50 or more problematic paragraphs. Users can still publish translations with less than 50 problematic paragraphs, but translations with 10 to 49 problematic paragraphs will be added to a tracking category of potentially unreviewed translations for the community to review.


 * Previous deleted translations. For users with some translations deleted in the last 30 days, the limits will be much more strict to prevent recurring problems. In those cases, translations with 10 problematic paragraphs or more will be prevented from publishing, while those with 9 or less problematic paragraphs will be added to a tracking category of potentially unreviewed translations for the community to review.

This will provide a way to accommodate cases where the automatic translation was exceptionally good, but still avoid potential abuse of the feature (i.e., not following blindly the user confirmation).
 * User confirmation. A less strict threshold is considered for paragraphs that users marked as resolved, as a signal that the user reviewed and confirmed the status of the translation. For paragraphs where the unmodified content warning was shown but the user marked it as resolved, we  apply a less strict threshold (accepting 95% of Machine translation or 75% of source content).

Contents not affected by the limits
Some contents are not expected to be edited significantly, and they are not considered when applying the limits described above.

Very short section titles, citations, or the list of references are excluded from the checks.

Otherwise, users may get misleading warnings because of contents such as the book titles in their references that they were not expected to translate.

Adjusting the limits
The limits described above provide a set of general mechanisms, but they may need adjustments to the particular needs of each wiki.

Based on initial evaluations, the amount of modifications needed to the initial machine translation can range from 10% to 70% depending on the language pair.

On some wikis the default limits may be too strict, generating unnecessary noise or preventing perfectly valid translations from publishing.

On other wikis, the limits may not be strict enough, allowing the publication of translations that are not edited enough.

Adjusting the different thresholds allows to make these limits more or less strict according to the needs of each wiki.

Feedback from native speakers is essential to adjust the limits properly.

If the current limits don't seem to work well based on your experience creating or reviewing translations, please share your feedback and we can explore how to better adjust them.

When providing feedback about adjusting the thresholds we recommend trying to create several example translations (make sure to check the publishing options if your test is not intended to be published as regular content).

When testing how the limits work for your language, it is useful to keep in mind the following:


 * Check for both cases. Make sure to check how the limits work for translations where the content has not been edited enough and also for those where the initial translation has been edited enough. In this way you can more easily find the right balance for the limits.  Checking only one type of problem can lead you to suggest moving the thresholds too far in the opposite direction.


 * Check different contents. Content in a wiki is very diverse, and machine translation may work much better for some cases compared to others. For example, content full of numeric data or technical names may require less editing by users than content with more descriptive text.  Make sure to test with different articles, making translations of different lengths and involving different types of content.

In any case, after each change further testing may be needed to verify the improvements.
 * Prepare to iterate. Adjusting the thresholds is an iterative process. It may require to make a custom adjustment to the thresholds or improve the general approach.

Adjusting the limits in collaboration with editors has proven to be effective.

For example, initial results show that the Indonesian community has reduced significantly the number of problematic translations by restricting the publication of translations with more than 70% of unmodified machine translation.

Similar adjustments have been made for Telugu and Assamese

There is no automatic tool that is infallible, and these limits are not an exception.

The process of content review by the community is still essential, but these limits provide communities with tools to reduce the number of translations they have to focus on, making the review process much more effective.

Please share your feedback and we can explore how to better adjust them.

Tracking potentially unreviewed translations
A tracking category with the name "cx-unreviewed-translation-category" is provided for the community to easily find the articles that were published with some content exceeding the recommended limits.

You can find this category in the list of tracking categories on each wiki.

There you can find the articles that passed the limits that prevent publishing but had still some paragraphs that have been edited less than expected.

For example the Indonesian category includes articles that have less than 40% of machine translation overall but have some paragraphs with more than 80% of unmodified machine translation.

Measuring translation quality
Evaluating content quality automatically is not trivial.

Deletion ratios provide a useful estimation on whether the content created was good-enough for the community of editors not to delete it.

Based on the analysis of deletion ratios, articles that are created as translations are less likely to be deleted compared to an initial version of the article created from scratch.

This suggests that it may not be practical to set the limits for participation by translating much higher than those set for other ways of article creation.

Content translation adds a contenttranslation edit tag to the published translations, for communities to be able to focus on the translations created with the tool using Recent changes and similar tools.

In addition, 1>Special:MyLanguage/Content translation/Published translations|data on published translations and [https://language-reportcard.wmflabs.org/cx2/#mt-engines the statistics for machine translation use] are available for anyone to analyze.

Other limits based on user expertise


Some wikis have implemented other restrictions for translating based on the user rights as a way to reduce the creation of low quality translations.

For example, English Wikipedia requires users to be extended confirmed, which means they need to make 500 edits on English Wikipedia before they are allowed to publish a translation as an article.

Newer editors can still publish translated articles in the  or   namespaces, and then move the article to the mainspace.

This restriction was created before the system of limits described in this page was available, and it is not the recommended approach to encourage the creation of good quality translations.

Before adding restrictions that do not take into account the content created, consider going through the process of adjusting the limits of unmodified content as described above.

The limits can be made as strict as needed to prevent low quality translations, while still allowing editors making good translations to publish them.