Continuous integration/Zuul/gearman upgrade

notes dropped by Antoine "hashar" Musso to prepare the migration of Zuul to a version using Gearman

Preparation

Make sure Zuul gearman port is firewalled out and only access from localhost Gerrit change 94136 Done
Python dependencies are installed on gallium Gerrit change 93454 Done
Verify the configuration files contains gearman configuration Gerrit change 93457 Done
python-voluptuous version is below 0.7.x Done

Upgrading

shutdown Zuul (sudo /etc/init.d/zuul stop) Done
bump master branch to the version in labs: git push -f gerrit labs:master Done
tag new version (tagged: wmf-deploy-20131120) Done
make sure the branch has the dependency hack Gerrit change 93452 (it is in labs branch already anyway). Done
on gallium update the code as root:

cd /usr/local/src/zuul
git remote update
git reset --hard origin/master
git tag wmf-deploy-`date +%Y%m%d` HEAD
git push --tags

Done

Then actually install the new version:

HTTP_PROXY=. HTTPS_PROXY=. python setup.py install

If it fails, it is surely because a python dependency is missing or the source code is missing the hack Gerrit change 93452. python-voluptuous should be below 0.7 since we don't have a later version in our apt repository yet. Done

Minor issue Antoine had to manually install the python-pip package to be able to run the install. Package removed after upgrade.

Restarting

Update Zuul layout configuration by merging in Gerrit change 93590 and pulling in /etc/zuul/wikimedia as user jenkins. Done

Minor issue had to delete old versions because pip did not found the new version but was still relying on the previous one. Aka: rm -fR /usr/local/lib/python2.7/dist-packages/zuul*. Result:

 $ zuul-server --version
 Zuul version: 6241272

Validate the layout using the new code base: Done

zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/wikimedia/layout.yaml -t

If all went fine, start Zuul!! Done

sudo /etc/init.d/zuul start

Look at the logs in /var/log/zuul/zuul.log or /var/log/zuul/debug.log

Enable Gearman

Enable Gearman in Jenkins https://integration.wikimedia.org/ci/configure then check 'Enable Gearman'. It needs to point to 127.0.0.1:4370. The Test Connection button would confirm a connection to Zuul daemon.

Trigger a job. Done

~~Success!~~FAILURE :-(

Restropective to be posted on wikitech-l. Basically the Jenkins gearman plugin triggers job on any slave, regardless of their jobs restrictions. That is an upstream bug.