Continuous integration/Zuul/gearman upgrade

notes dropped by Antoine "hashar" Musso to prepare the migration of Zuul to a version using Gearman

Preparation

 * Make sure Zuul gearman port is firewalled out and only access from localhost ✅
 * Python dependencies are installed on gallium ✅
 * Verify the configuration files contains gearman configuration ✅
 * python-voluptuous version is below 0.7.x ✅

Upgrading

 * shutdown Zuul (sudo /etc/init.d/zuul stop) ✅
 * bump  branch to the version in labs:  git push -f gerrit labs:master ✅
 * tag new version (tagged: wmf-deploy-20131120) ✅
 * make sure the branch has the dependency hack (it is in   branch already anyway). ✅
 * on gallium update the code as root:

cd /usr/local/src/zuul git remote update git reset --hard origin/master git tag wmf-deploy-`date +%Y%m%d` HEAD git push --tags

✅

Then actually install the new version:

HTTP_PROXY=. HTTPS_PROXY=. python setup.py install

If it fails, it is surely because a python dependency is missing or the source code is missing the hack. python-voluptuous should be below 0.7 since we don't have a later version in our apt repository yet. ✅

Minor issue Antoine had to manually install the  package to be able to run the install. Package removed after upgrade.

Restarting
Update Zuul layout configuration by merging in  and pulling in /etc/zuul/wikimedia as user jenkins. ✅

Minor issue had to delete old versions because pip did not found the new version but was still relying on the previous one. Aka:. Result:

$ zuul-server --version Zuul version: 6241272

Validate the layout using the new code base: ✅

zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/wikimedia/layout.yaml -t

If all went fine, start Zuul!! ✅

sudo /etc/init.d/zuul start

Look at the logs in /var/log/zuul/zuul.log or /var/log/zuul/debug.log

Enable Gearman
Enable Gearman in Jenkins https://integration.wikimedia.org/ci/configure then check 'Enable Gearman'. It needs to point to 127.0.0.1:4370. The Test Connection button would confirm a connection to Zuul daemon.

Trigger a job. ✅

Success! FAILURE :-(

Restropective to be posted on wikitech-l. Basically the Jenkins gearman plugin triggers job on any slave, regardless of their jobs restrictions. That is an upstream bug.