Content translation/Deployments/How-to

This document describes deployment procedure for ContentTranslation, cxserver, Apertium and OpusMT.

ContentTranslation

Content Translation is updated via regular MediaWiki train. In case of manual update needed,

1. Use Backport window to Cherry-pick desired changes.
2. Make sure that Gerrit patch is merged only "after" deployment server is updated to the branch we want to deploy.

See also

1. Branches at Gerrit interface: https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ContentTranslation,branches
2. To manually update extension branch: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule (You'll need clean copy of Mediawiki/core)

cxserver

Testing

Note image tag version from Gerrit patch to be deploy. For eg: gerrit:502964 has 2019-04-11-112002-production tag.

Run it:

docker run -p 4000:8080 docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-04-11-112002-production --it --entrypoint /bin/bash -c config.dev.yaml

Where, config.dev.yaml is local cxserver config file.

Endpoints can be tested at: http://localhost:4000

Config files

cxserver on Production config stays in:

helmfile.d/services/cxserver/values.yaml and,

WMF specific config stays in:

deployment-charts/charts/cxserver/templates/_config.yaml

When config.yaml is updated. Update cxserver version in deployment-charts/charts/cxserver/Chart.yaml

Also see

• Upgrade cxserver charts described at: https://gerrit.wikimedia.org/g/operations/deployment-charts
• helm can be install using methods described at: https://helm.sh/docs/intro/install/

Deployment

1. Clone deployment-charts repository (for first time).
2. Do needful changes in config (update image or other configuration changes as needed).
3. Make a CR (Example: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/623475) and after a successful review, merge it.
4. After merge, log in in a deployment server (eg: deploy1001), there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
5. Go to /srv/deployment-charts/helmfile.d/services/cxserver.
6. Execute: helmfile -e ${CLUSTER} diff This will show the changes that it will be applied on the cluster.
7. Execute: helmfile -e ${CLUSTER} -i apply This will materialize the previous diff in the cluster and also will log into SAL the change.

Secrets

If secrets like API key or token need update, it need to be done via SRE at Private Puppet repository.

To update or new keys, open Phabricator task with details and subscribe SRE clinic duty person. Example: task T284887

Status

This is done using helmfile:

1. Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver on a deployment server
2. Unless you are mid un-applied changes the current values files should reflect the deployed values
3. You can check for un-applied changes with: helmfile -e ${CLUSTER} diff
4. You can see the status with helmfile -e ${CLUSTER} status

Logs

• cxserver logs are available in logstash as 'cxserver-last-24-hours' dashboard.
• Logs can be access from deploy1002 if needed:

kube_env cxserver eqiad

kubectl logs cxserver-production-6c4f65bc-z6hcb cxserver-production

cxserver-production-6c4f65bc-z6hcb is pod name.

To see all logs:

kubectl logs -l app=cxserver -c cxserver-production

Rolling back changes

If you need to roll back a change because something went wrong:

1. Revert the git commit to the deployment-charts repository.
2. Merge the revert (with review if needed)
3. Wait one minute for the cron job to pull the change to the deployment server
4. Change directory to /srv/deployment-charts/helmfile.d/services/cxserver
5. Execute helmfile -e ${CLUSTER} diff to see what you'll be changing.
6. Execute helmfile -e ${CLUSTER} -i apply where CLUSTER is one of (staging, eqiad, codfw).

When patch with chart reverted, helmfile will pick highest number of chart present. Reverting such change will require pinning desired chart after revert configuration. eg. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/803873

Updating Template Parameter Alignment database

Main page: Content_translation/Deployments/How-to/TPA

Database

• x1: wikishared.cx_*: cxserver main database.
• m5-master: titles: section title mapping database.

Access

x1

x1 can be access via sql.php script on mwmaint server. See: https://wikitech.wikimedia.org/wiki/Debugging_in_production#Debugging_databases

Note that ContentTranslation in testwiki is not using wikishared and using separate database testwiki.

m5-master

m5-master requires cxserver user password access.

% mysql --skip-ssl -h m5-master.eqiad.wmnet -ucxserver -p
Enter password:

Also see

• Deploying with helmfile
• List of cxserver Production tags
• cxserver Grafana dashboard
• Kubernetes at Wikitech
• Minikuke (ie Kubernetes on laptop)

OpusMT

OpusMT service runs on opusmt.wmflabs.org cloud instance.

service

To see status of the service,

sudo service opusmt status

To restart the service,

sudo service opusmt restart

systemd service file

[Unit]
Description=OpusMT service 
Documentation=https://github.com/Helsinki-NLP/Opus-MT 
After=network.target 

[Service]
Type=simple 
ExecStart=/usr/local/bin/opusmt-server 
Restart=on-failure 

[Install]
WantedBy=default.target

Add new language support

1. Follow documentation https://wikitech.wikimedia.org/wiki/User:Santhosh/OpusMT_Setup#Opus_MT to download new language pair and setup on the instance.
2. Update language pair at: https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/config/OpusMT.yaml and submit patch for review. This requires deployment of cxserver in production via deployment-charts.
3. Make sure to restart opusmt service.