Content translation/Deployments/How-to

This document describes deployment procedure for ContentTranslation, cxserver and Apertium.

ContentTranslation[edit]

Content Translation is updated via regular MediaWiki train. In case of manual update needed, follow the steps below.

Update desired branches to desired commit/update using delete and create branches at Gerrit interface, https://gerrit.wikimedia.org/r/#/admin/projects/mediawiki/extensions/ContentTranslation,branches In case of emergency, note down old hash.
Read and follow: https://wikitech.wikimedia.org/wiki/How_to_deploy_code#Updating_the_submodule Make sure you've clean copy of MediaWiki/core for this.
You're ready for actual deployment.

cxserver[edit]

Testing[edit]

Note image tag version from Gerrit patchset to be deploy. For eg: https://gerrit.wikimedia.org/r/#/c/mediawiki/services/cxserver/+/502964/ has 2019-04-11-112002-production tag.

Run it:

docker run -p 4000:8080 docker-registry.wikimedia.org/wikimedia/mediawiki-services-cxserver:2019-04-11-112002-production --it --entrypoint /bin/bash -c config.dev.yaml

Where, config.dev.yaml is local cxserver config file.

Endpoints can be tested at: http://localhost:4000

Config files[edit]

cxserver on Production config stays in:

helmfile.d/services/cxserver/values.yaml and,

WMF specific config stays in:

deployment-charts/charts/cxserver/templates/_config.yaml

When config.yaml is updated. Update cxserver version in deployment-charts/charts/cxserver/Chart.yaml

and then run,

helm package cxserver

From charts/ directory, the repository index should then be rebuilt using:

helm repo index .

Submit generated tar file and index as Gerrit review.

Also see[edit]

Upgrade cxserver charts described at: https://gerrit.wikimedia.org/g/operations/deployment-charts
helm can be install using methods described at: https://helm.sh/docs/intro/install/

Deployment[edit]

Clone deployment-charts repository (for first time).
Do needful changes in config (update image or other configuration changes as needed).
Make a CR (Example: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/623475) and after a successful review, merge it.
After merge, log in in a deployment server (eg: deploy1001), there is a cron (1 minute) that will update the /srv/deployment-charts directory with the contents from git.
Go to /srv/deployment-charts/helmfile.d/services/cxserver.
Execute: helmfile -e ${CLUSTER} diff This will show the changes that it will be applied on the cluster.
Execute: helmfile -e ${CLUSTER} -i apply This will materialize the previous diff in the cluster and also will log into SAL the change.

Secrets[edit]

If secrets like API key or token need update, it need to be done via SRE at Private Puppet repository.

Status[edit]

This is done using helmfile:

Change directory to /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver on a deployment server
Unless you are mid un-applied changes the current values files should reflect the deployed values
You can check for un-applied changes with: helmfile -e ${CLUSTER} diff
You can see the status with helmfile -e ${CLUSTER} status

Logs[edit]

cxserver logs are available in logstash as 'cxserver-last-24-hours' dashboard.
Logs can be access from deploy1001 if needed:

cd /srv/deployment-charts/helmfile.d/services/${CLUSTER}/cxserver

source .hfenv

kubectl logs cxserver-production-6c4f65bc-z6hcb cxserver-production

cxserver-production-6c4f65bc-z6hcb is pod name.

To see all logs: kubectl logs -l app=cxserver -c cxserver-production

Rolling back changes[edit]

If you need to roll back a change because something went wrong:

Revert the git commit to the deployment-charts repo
Merge the revert (with review if needed)
Wait one minute for the cron job to pull the change to the deployment server
Change directory to /srv/deployment-charts/helmfile.d/services/cxserver
Execute helmfile -e ${CLUSTER} diff to see what you'll be changing.
Execute helmfile -e ${CLUSTER} -i apply where CLUSTER is one of (staging, eqiad, codfw).