Requests for comment/Reevaluate LocalisationUpdate extension

Tracked in Phabricator
Task T158360

Has LocalisationUpdate had its time?

Request for comment (RFC)

Reevaluate LocalisationUpdate extension
Component	General
Creation date	16 February 2017
Author(s)	Reedy, ^demon
Document status	declined See Phabricator.

Background

The LocalisationUpdate extension comes from a time when MediaWiki deployments were few and far between. Updates to interface messages were committed to master (or trunk) and we only branched and deployed from master at most a few times a year.

Over the past few years (mostly since the Scap rewrites), this hasn't been the case. We now deploy from master of MediaWiki core and extensions on a weekly basis.

Use case

The original (and still current) use cases for the existence of LocalisationUpdate:

• Translations must reach the live wikis quickly (definitely within a week), so that translators see the effect of their work and users always have the best translation on all wikis.
  • When translations take multiple days to be applied to the wikis, what typically happens is that local administrators start applying the translations locally and eventually the translations diverge and have to be cleaned up with costly processes (cf. T45917). If the delay becomes systematic, translators give up on translating important strings and instead take care only about the wikis where they're administrators, hence not benefiting other wikis and MediaWiki in general.
  • Such issues emerge very quickly: a list of new cases easily finds examples.

Current problems

LocalisationUpdate and Scap conflict

Each Tuesday a branch is cut from master and deployed using scap. Other subsequent deployments throughout the week can also involve scap.

LocalisationUpdate deploys updates nightly by importing message changes from the master branch, but its effect is only visible until the next scap run. Scap always re-deploys the messages as they are in the current week's wmf branch. The old versions are then visible until LocalisationUpdate runs again the following evening.

Unmanned deploy

Deployments by LocalisationUpdate happen at a time when not many people are around. There is the potential of staged but not-yet-deployed changes on WMF deployment servers being deployed unknowingly.

This is somewhat mitigated by the fact l10nupdate uses its own directory structure to work, not /srv/mediawiki-staging (but still, it's theoretically possible someone populates these directories).

Unmaintained

The extension has often been broken for days or weeks due to a lack of maintenance and ownership. No one is responsible for the code, those who have to support it (Release Engineering) do not have an adequate understanding of it.

Independent of Scap

It is triggered by legacy shell scripts that, unlike all other deployment tools, are not part of Scap.

File system issues

Uses a weird mix of www-data and l10nupdate for file ownership in its directories on various hosts. Makes cleanup of older branches more difficult.

Doesn't always run

Translators expect to see their translations the next day, also to fit their "working" schedules as volunteers (e.g. translating on Saturday and verifying the effect on Sunday to make any correction).

However l10nupdate doesn't always run as expected, either for unlogged/unidentified issues with the deployment system or for intentional decrease of service. The result is that volunteer translator may be unable to add or correct translations on time (e.g. for multiple days after a deploy of new code with new messages, or for a week if they can only translate on weekends).

The deployment systems should be made robust enough to handle the level of service that we need.

Proposals

1. Make sure LocalisationUpdate works consistently and translations are deployed within 24 hours from being committed.
2. Give up on a general solution for the use case and accept a permanently decreased service with periodical cleanups needed:
   • Disable and completely remove LocalisationUpdate from all WMF wikis;
   • Allow SWAT windows to do scap deploys of new/updated messages where these updates are required for events, or specific launch of a feature ahead of the general train deploy;
3. Set up a system to identify diverging/redundant translations and fix them automatically through git commits to wmf branches that are deployed with Scap by SWAT.