Continuous integration/Architecture/Troubleshooting

The continuous integration infrastructure is a machine with many moving parts. As with any machine of sufficient complexity, sometimes things go wrong. This page amalgamates information on troubleshooting the continuous integration infrastructure so that when things go wrong there is one unified place to go.

Ideally, any troubleshooting tips, tricks, or diagnostic measures should be included in this page. Effectively, this page is a CI Troubleshooting Cheatsheet.

This page is divided up into infrastructure pieces, where those live, what the known issues are with those pieces, and first-step solutions to common problems.

Stuck Nodes
From time-to-time Jenkins nodes will get stuck waiting for executors even though there is nothing running on those machines.

There are 2 possible issues Jenkins executioner lock or a Gearman Deadlock

Jenkins executioner lock

 * Take node offline in Jenkins
 * Kill any jenkins jobs running on the node via Jenkins UI
 * Kill all pending jobs in the Jenkins queue that are "waiting on executors"
 * Disconnect the node
 * Bring node back online (button labeled "Bring this node back online")
 * Launch slave agent (there's a button that says this)
 * Check agent log to see that it connected

Sometimes you have to do this whole dance several times before Jenkins realizes that the there are a bunch of executors that it can use.

Gearman Deadlock

 * Go to https://integration.wikimedia.org/ci/manage
 * Search page for "Enable Gearman"
 * Un-check the checkbox
 * Save
 * Wait 30s
 * Check the "Enable Gearman" checkbox
 * Save

This second method may interrupt communication between running Jenkins jobs and Zuul but it seems to work even when the offline/online method fails to clear the deadlock.

Don'ts

 * Don't restart gallium.wikimedia.org— in the   column, this may be broken. Also   will probably show timeouts waiting for image deletion.

The common fix is finding someone in #wikimedia-labs to restart RabbitMQ. Instance creation via horizon/wikitech is broken as well as nodepool instance deletion. To verify a fix, try:

nodepool delete --now [instance-id-marked-for-deletion]

Don'ts
¯\_(ツ)_/¯