Jump to content

Continuous integration/Zuul/gearman upgrade

From mediawiki.org

notes dropped by Antoine "hashar" Musso to prepare the migration of Zuul to a version using Gearman

Preparation

[edit]
  • Make sure Zuul gearman port is firewalled out and only access from localhost Gerrit change 94136 Yes Done
  • Python dependencies are installed on gallium Gerrit change 93454 Yes Done
  • Verify the configuration files contains gearman configuration Gerrit change 93457 Yes Done
  • python-voluptuous version is below 0.7.x Yes Done

Upgrading

[edit]
  • shutdown Zuul (sudo /etc/init.d/zuul stop) Yes Done
  • bump master branch to the version in labs: git push -f gerrit labs:master Yes Done
  • tag new version (tagged: wmf-deploy-20131120) Yes Done
  • make sure the branch has the dependency hack Gerrit change 93452 (it is in labs branch already anyway). Yes Done
  • on gallium update the code as root:
cd /usr/local/src/zuul
git remote update
git reset --hard origin/master
git tag wmf-deploy-`date +%Y%m%d` HEAD
git push --tags

Yes Done

Then actually install the new version:

HTTP_PROXY=. HTTPS_PROXY=. python setup.py install

If it fails, it is surely because a python dependency is missing or the source code is missing the hack Gerrit change 93452. python-voluptuous should be below 0.7 since we don't have a later version in our apt repository yet. Yes Done

Minor issue Antoine had to manually install the python-pip package to be able to run the install. Package removed after upgrade.

Restarting

[edit]

Update Zuul layout configuration by merging in Gerrit change 93590 and pulling in /etc/zuul/wikimedia as user jenkins. Yes Done

Minor issue had to delete old versions because pip did not found the new version but was still relying on the previous one. Aka: rm -fR /usr/local/lib/python2.7/dist-packages/zuul*. Result:

 $ zuul-server --version
 Zuul version: 6241272


Validate the layout using the new code base: Yes Done

zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/wikimedia/layout.yaml -t

If all went fine, start Zuul!! Yes Done

sudo /etc/init.d/zuul start

Look at the logs in /var/log/zuul/zuul.log or /var/log/zuul/debug.log

Enable Gearman

[edit]

Enable Gearman in Jenkins https://integration.wikimedia.org/ci/configure then check 'Enable Gearman'. It needs to point to 127.0.0.1:4370. The Test Connection button would confirm a connection to Zuul daemon.

Trigger a job. Yes Done


Success!FAILURE :-(

Restropective to be posted on wikitech-l. Basically the Jenkins gearman plugin triggers job on any slave, regardless of their jobs restrictions. That is an upstream bug.