Continuous integration/Zuul/gearman upgrade

notes dropped by Antoine "hashar" Musso to prepare the migration of Zuul to a version using Gearman

Preparation

 * Make sure Zuul gearman port is firewalled out and only access from localhost ✅
 * Python dependencies are installed on gallium ✅
 * Verify the configuration files contains gearman configuration ✅
 * python-voluptuous version is below 0.7.x ✅

Upgrading

 * shutdown Zuul (sudo /etc/init.d/zuul stop) ✅
 * bump master branch to the version in labs: git push -f gerrit labs:master ✅
 * tag new version (tagged: wmf-deploy-20131120) ✅
 * make sure the branch has the dependency hack (it is in labs branch already anyway). ✅
 * on gallium update the code as root:

cd /usr/local/src/zuul git remote update git reset --hard origin/master git tag wmf-deploy-`date +%Y%m%d` HEAD git push --tags

✅

Then actually install the new version:

HTTP_PROXY=. HTTPS_PROXY=. python setup.py install

If it fails, it is surely because a python dependency is missing or the source code is missing the hack. python-voluptuous should be below 0.7 since we don't have a later version in our apt repository yet. ✅

Minor issue Antoine had to manually install the python-pip package to be able to run the install. Package removed after upgrade.

Restarting
Update Zuul layout configuration by merging in  and pulling in /etc/zuul/wikimedia as user jenkins. ✅

Minor issue had to delete old versions because pip did not found the new version but was still relying on the previous one. Aka:  rm -fR /usr/local/lib/python2.7/dist-packages/zuul*. Result:

$ zuul-server --version Zuul version: 6241272

Validate the layout using the new code base: ✅

zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/wikimedia/layout.yaml -t

If all went fine, start Zuul!! ✅

sudo /etc/init.d/zuul start

Look at the logs in /var/log/zuul/zuul.log or /var/log/zuul/debug.log

Enable Gearman
Enable Gearman in Jenkins https://integration.wikimedia.org/ci/configure then check 'Enable Gearman'. It needs to point to 127.0.0.1:4370. The Test Connection button would confirm a connection to Zuul daemon.

Trigger a job. ✅

Success! FAILURE :-(

Restropective to be posted on wikitech-l. Basically the Jenkins gearman plugin triggers job on any slave, regardless of their jobs restrictions. That is an upstream bug.