Continuous integration/Zuul/gearman upgrade

notes dropped by Antoine "hashar" Musso to prepare the migration of Zuul to a version using Gearman

Preparation

edit
  • Make sure Zuul gearman port is firewalled out and only access from localhost Gerrit change 94136   Done
  • Python dependencies are installed on gallium Gerrit change 93454   Done
  • Verify the configuration files contains gearman configuration Gerrit change 93457   Done
  • python-voluptuous version is below 0.7.x   Done

Upgrading

edit
  • shutdown Zuul (sudo /etc/init.d/zuul stop)   Done
  • bump master branch to the version in labs: git push -f gerrit labs:master   Done
  • tag new version (tagged: wmf-deploy-20131120)   Done
  • make sure the branch has the dependency hack Gerrit change 93452 (it is in labs branch already anyway).   Done
  • on gallium update the code as root:
cd /usr/local/src/zuul
git remote update
git reset --hard origin/master
git tag wmf-deploy-`date +%Y%m%d` HEAD
git push --tags

  Done

Then actually install the new version:

HTTP_PROXY=. HTTPS_PROXY=. python setup.py install

If it fails, it is surely because a python dependency is missing or the source code is missing the hack Gerrit change 93452. python-voluptuous should be below 0.7 since we don't have a later version in our apt repository yet.   Done

Minor issue Antoine had to manually install the python-pip package to be able to run the install. Package removed after upgrade.

Restarting

edit

Update Zuul layout configuration by merging in Gerrit change 93590 and pulling in /etc/zuul/wikimedia as user jenkins.   Done

Minor issue had to delete old versions because pip did not found the new version but was still relying on the previous one. Aka: rm -fR /usr/local/lib/python2.7/dist-packages/zuul*. Result:

 $ zuul-server --version
 Zuul version: 6241272


Validate the layout using the new code base:   Done

zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/wikimedia/layout.yaml -t

If all went fine, start Zuul!!   Done

sudo /etc/init.d/zuul start

Look at the logs in /var/log/zuul/zuul.log or /var/log/zuul/debug.log

Enable Gearman

edit

Enable Gearman in Jenkins https://integration.wikimedia.org/ci/configure then check 'Enable Gearman'. It needs to point to 127.0.0.1:4370. The Test Connection button would confirm a connection to Zuul daemon.

Trigger a job.   Done


Success!FAILURE :-(

Restropective to be posted on wikitech-l. Basically the Jenkins gearman plugin triggers job on any slave, regardless of their jobs restrictions. That is an upstream bug.