Wikimedia Product/Wikimedia Product Infrastructure team/Service deployment process

Product Infrastructure maintains a number of Node.js services running on Kubernetes via the Wikimedia deployment pipeline. This page describes the process for deploying new versions of the services.

In a nutshell: when a patch in Gerrit is merged to the master branch of a service repo, a post-merge CI job is run to build a Docker image, according to the definition in the project's Blubberfile, from the updated master branch. Deploying the service simply means updating the service configurations on deployment.eqiad.wmnet and any Beta Cluster instances with the resulting image tag.

Bumping apt package versions
In order to achieve that, you need to:
 * Send an empty commit to trigger a new image build with the latest apt packages
 * Update deployment-charts accordingly
 * Follow the production deployment steps in the following section

Production
Helm charts defining the production service configurations are stored in the operations/deployment-charts repo. To deploy a new version to production, a patch must be created and merged updating the per-cluster service definitions under  with the Docker image tag created in the post-merge   job. Three deployment clusters are defined:,  , and   (with the latter two corresponding to Wikimedia's two core server clusters). An example patch updating each with a new Docker image tag for deployment is here. These patches can be self-merged by the deployer at deploy time. After merging, allow a minute or two for the repo on the production server to be synced with the master branch.

The next step is to sync the change to the Kubernetes cluster. On our production deployment servers (accessible as ) the deployment-charts repo lives at. Navigate to the service's helmfile directory for the staging cluster, examine a diff of the changes, and if the diff contains no surprises, sync the change: If everything looks OK, repeat the process in  and.

Beta Cluster
Deploying to the Beta Cluster involves a similar procedure. Log into Horizon, navigate to the deployment-prep project, select Compute > Instances from the left-hand nav menu, and click on the name of the instance to which you intend to deploy. In the instance view, click the Puppet Configuration tab and observe the Hiera Config section of the resulting screen. In the configuration shown, you should see a  setting containing a familiar-looking service-runner service configuration; find the   key under   and update it with your new image tag. The instance will pick up the updated configuration during the next Puppet run.