Content translation/Deployments/How-to

This document describes deployment procedure for ContentTranslation, cxserver and Apertium.

ContentTranslation
Content Translation is updated via regular MediaWiki train. In case of manual update needed, follow the steps below.


 * 1) Update desired branches to desired commit/update using delete and create branches at Gerrit interface, https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ContentTranslation,branches In case of emergency, note down old hash.
 * 2) Read and follow: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule Make sure you've clean copy of MediaWiki/core for this.
 * 3) You're ready for actual deployment.

Testing
Note image tag version from Gerrit patchset to be deploy. For eg: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/cxserver/+/502964/ has  tag.

Run it:

Where, config.dev.yaml is local cxserver config file.

Endpoints can be tested at: http://localhost:4000

Config files
cxserver on Production config stays in:

and,

WMF specific config stays in:

When config.yaml is updated. Update cxserver version in

and then run,

From charts/ directory, the repository index should then be rebuilt using:

Submit generated tar file and index as Gerrit review.

Also see

 * Upgrade cxserver charts described at: https://gerrit.wikimedia.org/g/operations/deployment-charts
 * helm can be install using methods described at: https://helm.sh/docs/intro/install/

Deployment

 * 1) Clone deployment-charts repository (for first time).
 * 2) Do needful changes in config (update image or other configuration changes as needed).
 * 3) Make a CR (Example: https://gerrit.wikimedia.org/r/#/c/528618/) and after a successful review, merge it.
 * 4) After merge, log in in a deployment server (eg: deploy1001), there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
 * 5) Go to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver where CLUSTER is one of (staging, eqiad, codfw).
 * 6) Execute:   This will show the changes that it will be applied on the cluster.
 * 7) Execute:   This will materialize the previous diff in the cluster and also will log into SAL the change.

Secrets
If secrets like API key or token need update, it need to be done via SRE at Private Puppet repository.

Status
This is done using helmfile:


 * 1) Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver on a deployment server
 * 2) Unless you are mid un-applied changes the current values files should reflect the deployed values
 * 3) You can check for un-applied changes with:
 * 4) You can see the status with

Logs

 * cxserver logs are available in logstash as 'cxserver-last-24-hours' dashboard.
 * Logs can be access from deploy1001 if needed:

cxserver-production-6c4f65bc-z6hcb is pod name.

To see all logs:

Rolling back changes
If you need to roll back a change because something went wrong:


 * 1) Revert the git commit to the deployment-charts repo
 * 2) Merge the revert (with review if needed)
 * 3) Wait one minute for the cron job to pull the change to the deployment server
 * 4) Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver where CLUSTER is one of (staging, eqiad, codfw).
 * 5) Execute   to see what you'll be changing.
 * 6) Execute

Apertium
Apertium deployment is done via updating apertium packages through Gerrit change and puppet.

Troubleshooting

 * 1) Check if package list are up-to-date in Puppet.
 * 2) Make sure to ask to restart 'apertium-apy' on scb1001 and scb1002 to any Ops member after adding new languages support for Apertium MT.
 * 3) Check logs at: