Content translation/Deployments/How-to

This document describes deployment procedure for ContentTranslation, cxserver, Apertium and OpusMT.

ContentTranslationEdit

Content Translation is updated via regular MediaWiki train. In case of manual update needed,

  1. Use Backport window to Cherry-pick desired changes.
  2. Make sure that Gerrit patch is merged only "after" deployment server is updated to the branch we want to deploy.

See alsoEdit

  1. Branches at Gerrit interface: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ContentTranslation,branches
  2. To manually update extension branch: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule (You'll need clean copy of Mediawiki/core)

cxserverEdit

TestingEdit

Note image tag version from Gerrit patchset to be deploy. For eg: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/cxserver/+/502964/ has 2019-04-11-112002-production tag.

Run it:

docker run -p 4000:8080 docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-04-11-112002-production --it --entrypoint /bin/bash -c config.dev.yaml

Where, config.dev.yaml is local cxserver config file.

Endpoints can be tested at: http://localhost:4000

Config filesEdit

cxserver on Production config stays in:

helmfile.d/services/cxserver/values.yaml and,

WMF specific config stays in:

deployment-charts/charts/cxserver/templates/_config.yaml

When config.yaml is updated. Update cxserver version in deployment-charts/charts/cxserver/Chart.yaml

and then run,

helm package cxserver

From charts/ directory, the repository index should then be rebuilt using:

helm repo index .

Submit generated tar file and index as Gerrit review.

Also seeEdit

DeploymentEdit

  1. Clone deployment-charts repository (for first time).
  2. Do needful changes in config (update image or other configuration changes as needed).
  3. Make a CR (Example: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/623475) and after a successful review, merge it.
  4. After merge, log in in a deployment server (eg: deploy1001), there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
  5. Go to /srv/deployment-charts/helmfile.d/services/cxserver.
  6. Execute: helmfile -e ${CLUSTER} diff This will show the changes that it will be applied on the cluster.
  7. Execute: helmfile -e ${CLUSTER} -i apply This will materialize the previous diff in the cluster and also will log into SAL the change.

SecretsEdit

If secrets like API key or token need update, it need to be done via SRE at Private Puppet repository.

To update or new keys, open Phabricator task with details and subscribe SRE clinic duty person. Example: task T284887

StatusEdit

This is done using helmfile:

  1. Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver on a deployment server
  2. Unless you are mid un-applied changes the current values files should reflect the deployed values
  3. You can check for un-applied changes with: helmfile -e ${CLUSTER} diff
  4. You can see the status with helmfile -e ${CLUSTER} status

LogsEdit

  • cxserver logs are available in logstash as 'cxserver-last-24-hours' dashboard.
  • Logs can be access from deploy1001 if needed:

cd /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver

source .hfenv

kubectl logs cxserver-production-6c4f65bc-z6hcb cxserver-production

cxserver-production-6c4f65bc-z6hcb is pod name.

To see all logs: kubectl logs -l app=cxserver -c cxserver-production

Rolling back changesEdit

If you need to roll back a change because something went wrong:

  1. Revert the git commit to the deployment-charts repo
  2. Merge the revert (with review if needed)
  3. Wait one minute for the cron job to pull the change to the deployment server
  4. Change directory to /srv/deployment-charts/helmfile.d/services/cxserver
  5. Execute helmfile -e ${CLUSTER} diff to see what you'll be changing.
  6. Execute helmfile -e ${CLUSTER} -i apply where CLUSTER is one of (staging, eqiad, codfw).

See alsoEdit

OpusMTEdit

OpusMT service runs on opusmt.wmflabs.org cloud instance.

serviceEdit

To see status of the service,

sudo service opusmt status

To restart the service,

sudo service opusmt restart

systemd service fileEdit

[Unit] 

Description=OpusMT service 
Documentation=https://github.com/Helsinki-NLP/Opus-MT 
After=network.target 

[Service] 
Type=simple 
ExecStart=/usr/local/bin/opusmt-server 
Restart=on-failure 

[Install] 
WantedBy=default.target

Add new language supportEdit

  1. Follow documentation https://wikitech.wikimedia.org/wiki/User:Santhosh/OpusMT_Setup#Opus_MT to download new language pair and setup on the instance.
  2. Update language pair at: https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/config/OpusMT.yaml and submit patch for review. This requires deployment of cxserver in production via deployment-charts.
  3. Make sure to restart opusmt service.