Content translation/Deployments/How-to

This document describes deployment procedure for ContentTranslation, cxserver, Apertium and OpusMT.

ContentTranslation
Content Translation is updated via regular MediaWiki train. In case of manual update needed,


 * 1) Use Backport window to Cherry-pick desired changes.
 * 2) Make sure that Gerrit patch is merged only "after" deployment server is updated to the branch we want to deploy.

Services
Status of all deployed services from Language team can be retrieved from this Grafana dashboard.

Deployment for all services are common except few minor changes. Following procedure can be apply for Apertium, Cxserver and MinT.

Testing
Note image tag version from Gerrit patch to be deploy. For eg: 502964 has  tag.

Run it:

Where, config.dev.yaml is local cxserver config file.

Endpoints can be tested at: http://localhost:4000

For example MinT can be tested using:

Config files
Production config stays in:

and,

WMF specific config stays in:

When anything under chart directory is updated. Bump chart version in

Also see

 * Upgrade of charts described at: https://gerrit.wikimedia.org/g/operations/deployment-charts
 * helm can be install using methods described at: https://helm.sh/docs/intro/install/

Deployment

 * 1) Clone deployment-charts repository (for first time).
 * 2) Do needful changes in config (update image or other configuration changes as needed).
 * 3) Make a CR (Example: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/623475) and after a successful review, merge it.
 * 4) After merge, log in in a deployment server (eg: deploy1001), there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
 * 5) Go to /srv/deployment-charts/helmfile.d/services/cxserver.
 * 6) Execute:   This will show the changes that it will be applied on the cluster.
 * 7) Execute:   This will materialize the previous diff in the cluster and also will log into SAL the change.

Status
This is done using helmfile:


 * 1) Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/SERVICE on a deployment server
 * 2) Unless you are mid un-applied changes the current values files should reflect the deployed values
 * 3) You can check for un-applied changes with:
 * 4) You can see the status with

Logs

 * Service logs are available in logstash such as 'cxserver-last-24-hours' and other similar dashboard.
 * Logs can be access from deploy1002 if needed:

cxserver-production-6c4f65bc-z6hcb is pod name.

To see all logs:

Rolling back changes
If you need to roll back a change because something went wrong:

Reverting patch

 * 1) Revert the git commit to the deployment-charts repository.
 * 2) Merge the revert (with review if needed)
 * 3) Wait one minute for the cron job to pull the change to the deployment server
 * 4) Change directory to /srv/deployment-charts/helmfile.d/services/SERVICE
 * 5) Execute   to see what you'll be changing.
 * 6) Execute    where CLUSTER is one of (staging, eqiad, codfw).

When patch with chart reverted, helmfile will pick highest number of chart present. Reverting such change will require pinning desired chart after revert configuration. eg. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/803873

Reverting to particular release
Sometime helm process is stuck in the 'pending-upgrade' status.

First, set deploy user permission by,

Check for last deployed release from REVISION using status command and rollback using,

Cxserver Database

 * x1: wikishared.cx_*: cxserver main database.
 * m5-master: titles: section title mapping database.

x1
x1 can be access via sql.php script on mwmaint server. See: https://wikitech.wikimedia.org/wiki/Debugging_in_production#Debugging_databases

Note that ContentTranslation in testwiki is not using wikishared and using separate database testwiki.

m5-master
m5-master requires cxserver user password access.

Secrets
If secrets like API key or token need update, it need to be done via SRE at Private Puppet repository.

To update or new keys, open Phabricator task with details and subscribe SRE clinic duty person. Example:

Also see

 * Deploying with helmfile
 * List of cxserver Cxserver Production tags
 * cxserver Grafana dashboard
 * Kubernetes at Wikitech
 * Minikuke (ie Kubernetes on laptop)