Wikimedia Release Engineering Team/Runbooks

This is a list of runbooks for the Wikimedia Release Engineering Team, covering step-by-step lists of what to do when things need doing, especially when things go wrong.

Gerrit

• Monitoring/Metrics
• Take a Thread dump
• Github replicas

GitLab

• Provisioning a new shared runner

Continuous integration

Configuration

• Add/modify CI for a new/existing repo (Zuul)
• Adding a new release pipeline for MediaWiki to CI (Zuul)
• Add/modify a new type of CI job (Jenkins Job Builder)
• Add/modify a new docker environment for CI jobs (Dockerfiles)
• Creating and deploying a new Quibble release (Dockerfiles + JJB for the critical CI workflow)
• Update doc.wikimedia.org static content (docroot)
• Replay a gerrit CI event into Zuul to re-trigger jobs

Infrastructure

• Clear part of Jenkins, when jobs are deadlocked ("waiting on executors") / Jenkins stuck
• Restart zuul (and drop all running jobs!)
• Agent remote call failed
• Upgrade Jenkins
• Work requests waiting in Zuul is CRITICAL -- this is usually a long chain of patchsets
  • If you caught it fast enough you can do: Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions
  • Otherwise your options are: wait or restart zuul
• Adding a new Jenkins agent
• Deploy doc.wikimedia.org changes
• Switch primary host for doc.wikimedia.org
• Switch primary host for integration.wikimedia.org

Phabricator

• Phabricator Administrative Commands
• Phabricator Deployment

Deployments Schedule

• Generating the wikitech:Deployments Page
• Generating the train blocking tasks on Phab