Continuous integration/Workflow 2014

The Workflow v1 (2012) is nearing completion. Time to take what we've learned, look at how the world has evolved and what new features are on the horizon. Here's what I hope we'll be doing before the end of 2014.

On submission of any patchset to any project on Gerrit by anyone:

  • Run its tests in a virtual machine
    • Gerrit notifies Zuul which distributes the job to a pool openstack of vagrant instances based on a pre-built image (these are teared down after a job and replaced with fresh ones automatically).
    • Which tasks to execute is specified in a declarative file inside the repository (e.g. a bit like .travis.yml, it would specify which of a handful of VM templates to use, and what commands to execute. such as npm install package.json & grunt test)
  • At the very least every repository runs lint checks (php lint, jshint, csslint, python pep8, puppet lint/validate)
  • For those opting in, also coding style checks (phpcs, jscs, python pep8)
  • MediaWiki core and extensions:
    • Install MediaWiki on local webserver (+ any extensions installed if relevant)
    • Run PHPUnit tests (optimised using phpunit-parallel)
    • Run QUnit tests
      • Run grunt/qunit (phantomjs)
      • Run grunt/karma (cross-browser qunit testing in saucelabs or browserstack)
    • Run selenium integration tests in saucelabs
    • Run pdiff-based job to detect any visual differences on common urls (not an assertion, purely informative to detect unintended visual side effects and to easily review intended visual differences).
    • Assert catch-all MediaWiki/php/apache error logs are empty
  • For operations/apache-config
    • Install the vhosts on the local instance and run integration tests to verify things work properly. Especially with regards to redirects which are easy to screw up (e.g. a rewrite case falling through and redirecting dozens of domains incorrectly, query string encoding etc.)
  • For operations/puppet
    • puppet validate

Tasks:

  • Install csslint
  • install jsc (bug 54218)
    • Set up jobs for it and start using it in core and selected extensions
  • Merge separate phpunit jobs back into one and use phpunit-parallel (bug 48217)
  • Track and assert error logs for a high-level catching of issues (bug 48002)
  • Implement virtual machine infrastructure
    • Set up one or more templates
    • Design and implement the travis-yml-like set up