Abstract Wikipedia team/Runbooks

This is a list of runbooks for the Abstract Wikipedia Team, covering step-by-step lists of what to do when things need doing, especially when things go wrong. scsd

Back-end services

 * Main page: Kubernetes/Deployments

Deploy a config update to the orchestrator

 * 1) Make a change to the orchestrator's helm chart in the deployment-charts repo in gerrit, make a commit, land it with a colleague or by yourself
 * 2) * Remember to bump in  the chart version number, so the change is picked up.
 * 3) Shell into production deployment server  and go to our service directory
 * 4) Check that the new change to deployment-charts git repo has made it automatically to the server
 * 5) Run   to validate that the helm chart applies and the diff looks correct
 * 6) Make a simple request via curl to check that the orchestrator performs as expected
 * 7) Run   to deploy the update to the Texas datacentre (the change is now live for some users)
 * 8) Run   to deploy the update to the Virginia datacentre (the change is now live for all users)
 * 9) Monitor production for a bit, and revert if needed

Deploy a new version of the orchestrator

 * 1) Make a change to the orchestrator repo in GitLab, make a Merge Request, wait for it to landed by a colleague (example)
 * 2) Make a config change to the orchestrator's helm chart as above, changing in   the   value of the docker image to the newly-created docker-registry tag from step 1. (example)

Disable an evaluator from being called

 * 1) Make a config change to the orchestrator's helm chart as above, changing in   the   value to remove the evaluator from the map of known evaluators.
 * 2) * If the evaluator you are removing is the only one assigned to that language, you are disabling evaluation in that language.

Add an evaluator to be called

 * 1)  Document how to stand up the config of the new evaluator
 * 2) Make a config change to the orchestrator's helm chart as above, changing in   the   value to add the evaluator to the map of known evaluators for the appropriate languages.
 * 3) * If the evaluator you are adding is the only one assigned to that language, you are enabling evaluation in that language.