This page is a translated version of the page Help:Content translation/Translating/Translation quality and the translation is 8% complete.
PD NB: At jo dizze side bewurkje, stimme jo yn om jo bydrage frij te jaan ûnder de CC0. Sjoch Helpsiden Publyk Domein foar mear ynformaasje. PD

As jo besykje oer te setten, is it baas en sjoch de ynhâld earst nei, ear't jo it publisearje. Jo wolle der wis fan wêze dat de ynhâld krekt itselde is as de oarspronklike betsjuttenis, en hifkje dat it natuerlik te lêzen is yn de doeltaal. De initial machine translation jout help en meitsje it proses fan oersetten flugger mei in brûkber begjinpunt, dochs it is baas dat de oersetters earst de tekst hifkje en bewurkje.

In ferskaat oan meganismen binne ornearre sa't oersetters de oarspronklike tekst krekt bewurkje kinne De oersettingsbewurker hâldt by hoefolle der fan de oarspronklike oersetting troch de meidogger oanpast is, en hat beskate limiten dy't: it publisearjen foarkomme, of warskôgje meidoggers en sette harren oan om de ynhâld te hifkjen.

Op dizze manier, jout it helpmiddel meidoggers de mooglikheid de ynisjele masine-oersetter goed brûke te kinnen, wylst der foarkommen wurdt dat it resultaat fan lege kwaliteit wurdt. Hjirûnder wurdt útlein hoe't dy limiten wurkje, hoe't se de taal oanpast wurde kinne en hoe't de kwaliteit fan de ynhâld mei de troch it helpmiddel makke oersettings beoardiele wurde kin.

Content translation tool measures the percentage of modifications that users make to the initial automatic translation provided. In this way, the system knows how many words have been added, removed, or modified from the initial translation. Measurements are made at two different levels: for each paragraph and for the whole translation. Different limits are applied at each level, as detailed below.

Limits for the whole translation

Error shown when trying to publish a translation with too much unmodified machine translation. This threshold was adjusted for Indonesian based on feedback from their editors.

Publication is blocked if 95% or more of the whole document consists of unmodified, machine translated content. This limit prevents near-raw machine translations and circumvents clear vandalism. It also prevents users from merely adding content, without editing the machine translation portion. As detailed below, this limit can be adjusted on a per-language basis.

Limits for each paragraph

Warning shown for a specific paragraph where unmodified machine translation exceeds the limits.

The percentage of user modifications is also measured for each paragraph. A paragraph is considered problematic when it contains more than 85% of the initial machine translation (or, when copying the contents from the source document, it contains more than 60% of unmodified content).

The translation editor will show a warning for each paragraph that is considered problematic, encouraging further edits by the user. In some cases, users are still able to publish, but the resulting page may get added to a tracking category of potentially unreviewed translations for the community to review. In other cases, users may not be allowed to publish at all.

The following are some of the factors considered for determining whether to allow users to publish or not (some of which are still in development):

  • The number of problematic paragraphs. Users are prevented from publishing translations with 50 or more problematic paragraphs.

Publication of translations with less than 50 problematic paragraphs is permitted, but those with 10 to 49 problematic paragraphs will be added to a tracking category of potentially unreviewed translations for the community to review.

  • Previous deleted translations. To prevent recurring problems, the tool identifies users whose published translations were deleted in the last 30 days, and imposes much more strict limits upon their subsequent translation efforts.

For users in this class, translations with 10 problematic paragraphs or more are prevented from publishing, while those with 9 or less problematic paragraphs are added to a tracking category of potentially unreviewed translations for the community to review.

  • User confirmation. A less strict threshold is considered for paragraphs that a user marks as resolved—taken as a signal that the user reviewed and confirmed the status of the translation.

For paragraphs where the unmodified content warning is shown, but the user marks it as resolved, a less strict threshold is applied (accepting 95% of Machine translation or 75% of source content). This will provide a way to accommodate cases where the automatic translation was exceptionally good, but still avoid potential abuse of the feature (i.e., not blindly following a user's confirmation).

Contents not affected by the limits

Some content is not expected to be edited significantly, and thus is not considered when applying the limits described above. Very short section titles, citations, or the list of references are excluded from review. Otherwise, users could receive misleading warnings about translating content that should not be, such as book titles appearing in references or other proper nouns.

Limits on the mobile experience

For the mobile experience the initial set of limits follow a simpler approach. At the moment, only the overall percentage of unmodified machine translation for the whole translation is considered. On mobile, the whole translation consist of just one section of the article.

In particular, a warning is shown when the percentage of unmodified machine translation is over 85% for the whole section, and publishing is prevented when the percentage of unmodified machine translation is over 95%.

Feedback on how the limits system work on the mobile context would be very useful to determine how to evolve this initial approach.

Adjusting the limits

The limits described above provide a set of general mechanisms, but they may need adjustment depending on the particular needs of each wiki. Based on initial evaluation, the amount of modification needed to initial machine translation can range from 10% to 70%, depending on the language pair. On some wikis, the default limits may be too strict, generating unnecessary noise or preventing perfectly valid translations from being published. On other wikis, the limits may not be strict enough, allowing the publication of translations that have not been edited enough.

Adjusting the different thresholds allows each wiki to tailor the tool's limits according to its particular needs. Feedback from native speakers is essential in properly adjusting the limits imposed. If the current limits don't seem to work well based on your experience in creating or reviewing translations, please share your feedback, and we can explore how to better adjust them.

When providing feedback about adjusting the thresholds, we recommend that you first create several example translations (make sure to check the publishing options if your test is not intended to be published as regular content). When testing how the limits work for your language, it is useful to keep in mind the following:

  • Check for both cases. Make sure to check how the limits work for both: translations where the content has not been edited enough, versus where it has been edited enough.

In this way, you can more easily find the right balance for the tool's limits feature. Checking only one type of problem can lead to moving the thresholds too far in the opposite direction.

  • Check different content. Content in our wikis is highly diverse, and machine translation may work much better for some cases compared to others.

For example, content that is full of numeric data or technical names may require less editing by users than content with more descriptive text. Make sure to test by translating of a variety of different article types, of varying lengths, with disparate content.

  • Prepare to iterate. Adjusting the thresholds is an iterative process.

It may require custom adjustments to the thresholds or that you improve your general approach. In any case, after each change, further testing may be needed to verify the improvements made.

Adjusting the limits in collaboration with editors has proven to be effective. For example, initial results show that the Indonesian community was able to significantly reduce the number of problematic translations they were receiving by restricting the publication of translations with more than 70% of unmodified machine translation content. Similar adjustments have been made for Telugu and Assamese language wikis. There is no automatic tool that is infallible, and these limits are not an exception.

The process of content review by the community is still essential, but these limits provide communities with a tool to reduce the number of translations they have to focus on, making the review process much more effective. Please share your feedback and we can explore how to better adjust them.

Tracking potentially unreviewed translations

A tracking category with the name "cx-unreviewed-translation-category" is provided for communities to easily find articles that have been published with some content exceeding the recommended limits.

You can find this category in the list of tracking categories on each wiki. Using it, you can track articles that passed the limits preventing publication, but that still had some paragraphs that were edited less than expected. For example the Indonesian Wikipedia's category includes articles that have less than 40% of machine translation overall, but which have some paragraphs with more than 80% of unmodified machine translation.

Measuring translation quality

Evaluating content quality automatically is not trivial. Deletion ratios provide a useful measure for estimating whether the content created was good enough for the community involved not to delete it. Based on the analysis of deletion ratios, articles that are created as translations are less likely to be deleted when compared with articles created from scratch. This suggests that it may not be practical to set the limits for participation through translating much higher than those set for other ways of article creation.

Find published translations

Content translation adds a contenttranslation edit tag to the published translations. This allows communities the ability to use Recent changes, and similar tools, to focus on pages created using the translation tool. In addition, data on published translations and the statistics for machine translation use are available for anyone to analyze.

Inspect a specific translation

Translation debugger example

The Translation debugger is a tool that allows the inspection of some metadata for a given translation, including the percentage of machine translation used for the whole document, and the translation service used for each paragraph. For specific types of content such as templates, the Content Translation Server API can be queried to check how templates will be transferred across languages.

Other limits based on user expertise

Error showing a publishing restriction based on the user expertise. This example is based on the decision of the English Wikipedia community to limit publishing directly to the mainspace to extended autoconfirmed users only.

Some wikis have implemented other restrictions for translating based on the user rights as a way to reduce the creation of low-quality translations. For example, English Wikipedia requires users to be extended confirmed, which means they need to make 500 edits on English Wikipedia before they are allowed to publish a translation as an article. Newer editors can still publish translated articles in the User: or Draft: namespaces, and then move the article to the mainspace.

This restriction was created before the system of limits described in this page was available, and it is not the recommended approach to encourage the creation of good quality translations.

Before adding restrictions that do not take into account the content created, consider going through the process of adjusting the limits of unmodified content as described above. The limits can be made as strict as needed to prevent low-quality translations, while still allowing publication by editors making good translations.