Continuous integration/Quibble

Quibble is a Python script for setting up a MediaWiki instance and running various tests against it.

It works by cloning MediaWiki core and several extensions, installing dependencies, creating the database, and running one or more test commands.

Further reading

• Introducing Quibble, Antoine Musso (April 2018), phabricator.wikimedia.org.
• https://doc.wikimedia.org/quibble/

Creating and deploying a new Quibble release

This is moderately fiddly and stressful, and should be done when you have a fair amount of time free and CI is relatively quiet; Fridays are commonly picked.

Creating the release

See RELEASE.rst in the repository.

Creating images

This covers the creation of Docker images using the Quibble git tag (example)

1. In integration-config's dockerfiles directory, manually edit the Dockerfile.template for the primary Quibble image to specify the new version (as of 2022-02-02, this is for quibble-buster). That is done in the pip3 install command line.
2. Use docker-pkg to create the appropriate cascade of changelog updates, e.g. docker-pkg -c dockerfiles/config.yaml --info update --reason "Bump quibble to 0.0.44" --version 0.0.44 quibble-buster dockerfiles/
3. Build the images locally to ensure they still build: docker-pkg -c dockerfiles/config.yaml --info build --use-cache dockerfiles/
4. Check the locally-built images to ensure that quibble correctly updated: ./dockerfiles/debug-image quibble-buster-php72 and then head /usr/local/bin/quibble and see that the correct version number is shown.
5. Create this update as a change, submit it to code review, and wait for it to be merged.
6. Create and publish the new images on the CI production server: ./fab deploy_docker (remember to !log this action in IRC) – 🐌 this will take a while, perhaps an hour or more

Update Jenkins jobs

Switch CI jobs over to the new image (example)

1. In integration-config's jjb directory, update the Jenkins job-builder YAML files to specify the new docker images: sed -i '' s/:0.0.42/:0.0.43/' jjb*
2. Verify all Quibble images have been updated: ./utils/docker-updates
3. Verify that these jobs build correctly: ./jjb-test -o output/
4. Get a list of all jobs which will be updated by the change; there will be around 150 as of 2020-06-04.
5. Push these updates as a commit to code review.
6. Based on your knowledge of the changes in quibble, pick a likely simple, infrequent, fast-running job to manually change over, and push it: ./jjb-update 'quibble-vendor-mysql-php72-docker'
7. Manually trigger run a run of this job through the Jenkins Web interface, and carefully watch the output to ensure it works as it used to where it should, and in a different way where Quibble's changes should change things.
8. If you spot an error, rollback the job to the definition in master immediately rather than debugging live; remember that other people's workflows depend on CI continuing to work.
9. Repeat this with increasingly major/high-profile jobs until you are satisfied that you or someone else on IRC would have noticed if the jobs were broken.
10. Update the rest of the jobs in your list.
11. Merge the commit as deployed.
12. You most probably want to verify that a recheck on mediawiki/core and mediawiki/extensions/Wikibase pass fully.
13. Continue to monitor CI for a while, in case things blow up despite your hard work. If it does, revert everything.

Reproducing CI runs locally

After you have pushed a change to Gerrit, CI will run and execute a number of jobs that run in Docker containers on the WMF infrastructure. If you have issues with the jobs on CI that you don't have when you run tests on your local setup, running the CI job on your machine can be a good test of whether the bug is reproducible or not. Once you have a bug reproduced on your local machine, it's usually only a matter of time before you can get to the bottom of what's causing the issue and fix it.

Follow the link from the Gerrit change to the CI output. The link should look something like:

https://integration.wikimedia.org/ci/job/quibble-composer-mysql-php74-noselenium/53/console

(though note that old CI builds logs disappear, so clicking exactly this link may show an empty log). From the log view, you should be able to find a Full Log link which will show you the complete console output for the job run. Somewhere in the first 10 seconds of log output, you should see a docker invocation to run the associated container:

13:05:00 + exec docker run --entrypoint=quibble-with-supervisord --tmpfs /workspace/db:size=320M --volume /srv/jenkins/workspace/quibble-composer-mysql-php74-noselenium/src:/workspace/src --volume /srv/jenkins/workspace/quibble-composer-mysql-php74-noselenium/cache:/cache --volume /srv/jenkins/workspace/quibble-composer-mysql-php74-noselenium/log:/workspace/log --volume /srv/git:/srv/git:ro --security-opt seccomp=unconfined --init --rm --label jenkins.job=quibble-composer-mysql-php74-noselenium --label jenkins.build=53 --env-file /dev/fd/63 docker-registry.wikimedia.org/releng/quibble-buster-php74:1.7.0-s1 --reporting-url=https://earlywarningbot.toolforge.org --packages-source composer --db mysql --db-dir /workspace/db --git-parallel=8 --reporting-url=https://earlywarningbot.toolforge.org --skip selenium,npm-test,phpunit-standalone,api-testing

The only information missing to run the same code on your machine is the environment variables (passed in CI as a file-descriptor) - these are available by clicking the Parameters link. Copy the build parameters to a local file and give it a name (like build-1234-quibble-composer-mysql-php74-noselenium.env):

EXT_DEPENDENCIES=mediawiki/extensions/AbuseFilter\nmediawiki/extensions/AntiSpoof\nmediawiki/extensions/ArticlePlaceholder\nmediawiki/extensions/BetaFeatures\nmediawiki/extensions/CentralAuth\nmediawiki/extensions/CheckUser\nmediawiki/extensions/CirrusSearch\nmediawiki/extensions/Cite\nmediawiki/extensions/CodeEditor\nmediawiki/extensions/ConfirmEdit\nmediawiki/extensions/DiscussionTools\nmediawiki/extensions/Echo\nmediawiki/extensions/Elastica\nmediawiki/extensions/EventBus\nmediawiki/extensions/EventLogging\nmediawiki/extensions/EventStreamConfig\nmediawiki/extensions/FlaggedRevs\nmediawiki/extensions/Flow\nmediawiki/extensions/GeoData\nmediawiki/extensions/GuidedTour\nmediawiki/extensions/Linter\nmediawiki/extensions/MobileApp\nmediawiki/extensions/MobileFrontend\nmediawiki/extensions/PageImages\nmediawiki/extensions/ParserFunctions\nmediawiki/extensions/PdfHandler\nmediawiki/extensions/Popups\nmediawiki/extensions/PropertySuggester\nmediawiki/extensions/Renameuser\nmediawiki/extensions/Scribunto\nmediawiki/extensions/SecurePoll\nmediawiki/extensions/SiteMatrix\nmediawiki/extensions/SyntaxHighlight_GeSHi\nmediawiki/extensions/TemplateData\nmediawiki/extensions/TextExtracts\nmediawiki/extensions/Thanks\nmediawiki/extensions/TimedMediaHandler\nmediawiki/extensions/TorBlock\nmediawiki/extensions/UniversalLanguageSelector\nmediawiki/extensions/UserMerge\nmediawiki/extensions/VisualEditor\nmediawiki/extensions/WikiEditor\nmediawiki/extensions/Wikibase\nmediawiki/extensions/WikibaseCirrusSearch\nmediawiki/extensions/WikibaseLexeme\nmediawiki/extensions/WikibaseLexemeCirrusSearch\nmediawiki/extensions/WikibaseMediaInfo\nmediawiki/extensions/WikibaseQualityConstraints\nmediawiki/extensions/WikimediaBadges\nmediawiki/extensions/WikimediaEvents\nmediawiki/extensions/cldr
ZUUL_BRANCH=REL1_39
ZUUL_PIPELINE=test-1_39
ZUUL_CHANGE_IDS=1012674,2
ZUUL_PATCHSET=2
SKIN_DEPENDENCIES=mediawiki/skins/MinervaNeue
EXT_NAME=Wikibase
ZUUL_URL=git://contint2002.wikimedia.org
ZUUL_REF=refs/zuul/REL1_39/Z1d0bc294151f47a9b376563a20275966
ZUUL_COMMIT=16705d3a2f8960d27f00b155ac5574d57fa7c501
ZUUL_CHANGES=mediawiki/extensions/Wikibase:REL1_39:refs/changes/74/1012674/2
ZUUL_CHANGE=1012674
ZUUL_PROJECT=mediawiki/extensions/Wikibase

Editing the docker command to point to your local environment file (setting the --env-file argument) should be all you need to do to trigger the execution of the same job locally on your machine, though note that any --volume mappings will be different on your machine and might need some preparation. For the purposes of local build reproduction the --label and --reporting-url arguments can be dropped / ignored.

Running Quibble Jobs locally

In the case of Quibble jobs, the steps for setting up your local environment are documented in the Quibble docs. Very briefly, to set up the folders required for the local volume mapping, run:

$ mkdir -p src && mkdir -p cache && mkdir -p log && mkdir -p ref
$ chmod o+w src cache log ref

in the folder that will be the working directory for your CI experiments. The src folder caches some context between executions of Quibble, so you might need to completely clear it out between executions. It might also be necessary to delete the src/LocalSettings.php file to convince Quibble to run the tests if you have used the --skip-deps or --skip-zuul arguments to Quibble.

Supposedly, making bare clones (git clone --bare) of the upstream git repositories inside the newly-created ref folder allows Quibble to avoid cloning the whole of Mediawiki and all extensions every time you run it. I'm not convinced that functionality is actually working as advertised (either that or I'm doing something wrong).

Attaching the Debugger

One of the massive advantages of running CI images locally is that you can (at least in principle) attach a debugger to the running tests to find out what's happening in the container. Unfortunately, Quibble docker images don't include XDebug. Fortunately, rebuilding the images is pretty straightforward.

Rebuilding the current docker image with XDebug support

The source for the docker images the integration-config repository. Clone the repository, and install the docker-pkg software to be able to make changes to the images. You may also want to install dch to be able to make Debian-format changelog updates (if not, you will have to make the changes manually with an editor).

$ sudo apt install devscripts    # optional
$ git clone https://gerrit.wikimedia.org/r/integration/config integration-config
$ cd integration-config
$ pipenv install docker-pkg
$ pipenv shell

Next, you will need to update the Dockerfile for the image that you are using. You will find a Dockerfile.template file in the dockerfiles folder corresponding to the image that you are working with. To enable XDebug, it suffices to add the line:

php7.4-xdebug

to the list of PHP modules (in the case of a PHP7.4 image). You will then need to bump the changelog so that docker-pkg notices that something has changed. If you have dch installed, simply run

$ dch -i -c changelog

in the folder of the Dockerfile that you have changed and enter a comment. Alternatively, edit the changelog by hand. Now you should be able to run:

$ cd ${workdir}/integration-config/dockerfiles
$ docker-pkg build .

and docker-pkg should build and tag a new version of the docker image for you. Note the version of this image - we will need that to run the new container.

Running the docker image with XDebug enabled

Once you have built the new docker image, you should be able to run it simply by changing the name of the image in the docker invocation. For example:

$ docker run --entrypoint=quibble-with-supervisord     \
             --tmpfs /workspace/db:size=320M \
             --volume "$(pwd)"/src:/workspace/src \
             --volume "$(pwd)"/cache:/cache \
             --volume "$(pwd)"/log:/workspace/log \
             --volume "$(pwd)"/ref:/srv/git:ro \
             --security-opt seccomp=unconfined \
             --env-file=env-wmf-quibble-vendor-mysql-php74-docker \
             --init --rm \
             docker-registry.wikimedia.org/releng/quibble-buster-php74:1.6.0-s6ubuntu1 \
             --packages-source composer \
             --db mysql --db-dir /workspace/db \
             --git-parallel=8 \
             --git-cache /srv/git/ \
             --phpunit-testsuite=extensions \
             --skip selenium,npm-test,phpunit-standalone,api-testing,phpbench,qunit

to run my modified 1.6.0-s6 image - dch adds a version suffix like ubuntu1 to locally-built versions by default.

Connecting with XDebug and PHPStorm

To connect to the running image with PHPStorm, you need to enable XDebug in the container and point it at the IDE. On Linux, you can add the lines:

XDEBUG_CONFIG=client_host=172.17.0.1 output_dir=/workspace/src
XDEBUG_MODE=develop,debug
PHP_IDE_CONFIG="serverName=Local Server"

to the --env-file that you are using to launch your docker image (replacing "Local Server" here with the name of the server you are about to create in PHPStorm). In PHPStorm, you will need to define a new local PHP Server so that PHPStorm can map the source files and find the breakpoints. The server should include a source mapping from /workspace/src to the src folder on your local machine / in the current working directory. You can also define an HTTP Server at this point - this will be handy if we later expose the Quibble HTTP Server for interactive debugging.

At this point you should be able to set a breakpoint in the debugger and launch Quibble and see that the run pauses when the breakpoint is hit. If you want to be very sure that the integration is working, you can set the XDEBUG_TRIGGER environment variable to any value, and the IDE should stop whenever a PHP script launches. Note that this will breakpoint for all composer invocations - you will need to hit play on your debugger a couple of times to make progress with the Quibble run.

Modifying local Quibble runs

Once you have the same failure on your local machine as you see in CI, you are probably going to want to make some changes to the code and see what difference it makes. Unfortunately, the Quibble container completely rebuilds the src folder on every run, and the container exits when the tests complete, so the window for making changes and interacting with the container is quite limited.

Introducing Quabble

To be able to play around with Quibble runs, it would be ideal if we could pause Quibble after it has completed all its setup steps, but before it runs any PHP / Node / QUnit / Selenium tests. Unfortunately, this isn't how Quibble works. Fortunately, Quibble is "just Python", and whereas the src folder is blown away every run, the cache folder is persistent so we can make changes there. To pause Quibble, we need to create two new files in the cache folder - quabble-with-supervisord and quabble.py:

quabble-with-supervisord

#!/bin/bash
set -euxo pipefail
/usr/bin/supervisord -c /etc/supervisor/supervisord.conf
exec /cache/quabble.py --web-backend=external --web-url=http://127.0.0.1:9413 "$@"

quabble.py

#!/usr/bin/python3
import logging
import time
import sys
from quibble.cmd import QuibbleCmd, get_arg_parser
import quibble

logging.basicConfig(level=logging.INFO)
logging.getLogger('quibble').setLevel(logging.INFO)

class WaitCommand:

    def __init__(self, delay):
        self.delay = delay

    def execute(self):
        time.sleep(self.delay)

    def __str__(self):
        return "WaitCommand sleeping for " + str(self.delay) + " seconds"


args = get_arg_parser().parse_args(sys.argv[1:])
if args.color:
    quibble.colored_logging()

cmd = QuibbleCmd()
project_dir, plan = cmd.build_execution_plan(args)
plan.append(WaitCommand(30000))
plan = [ step for step in plan if str(step).find("PHPUnit") == -1 and str(step).find("Qunit") == -1 ]

cmd.execute(
    plan,
    project_dir=project_dir,
    reporting_url=args.reporting_url,
    dry_run=args.dry_run,
)

Now we can modify our docker invocation to use our new script as the entrypoint (--entrypoint=/cache/quabble-with-supervisord):

$ docker run --entrypoint=/cache/quabble-with-supervisord     \
             --tmpfs /workspace/db:size=320M \
             --volume "$(pwd)"/src:/workspace/src \
             --volume "$(pwd)"/cache:/cache \
             --volume "$(pwd)"/log:/workspace/log \
             --volume "$(pwd)"/ref:/srv/git:ro \
             --security-opt seccomp=unconfined \
             --env-file=env-wmf-quibble-vendor-mysql-php74-docker \
             --init --rm \
             docker-registry.wikimedia.org/releng/quibble-buster-php74:1.6.0-s6ubuntu1 \
             --packages-source composer \
             --db mysql --db-dir /workspace/db \
             --git-parallel=8 \
             --git-cache /srv/git/ \
             --phpunit-testsuite=extensions \
             --skip selenium,npm-test,phpunit-standalone,api-testing,phpbench,qunit

After the source has been checked out and MediaWiki installed, the terminal should pause on the WaitCommand of Quabble.

Modifying the source code

You can modify the code in the src folder and the changes will be reflected live in the container. Note that the permissions on the files (nobody:nogroup) might prevent you making changes from your IDE, but you can either rsync changes over as root or edit files directly as root. Note that as soon as you restart Qu(i/a)bble, it will checkout all the sources from scratch. To avoid this, you need to use the --skip-zuul argument to Quibble, though in that case make sure to delete the LocalSettings.php that the installer has generated, otherwise Quibble will complain / abort.

Running tests manually

With Quabble running, you can launch a shell inside the docker image to run the CI steps individually. Run docker ps to see a list of running containers (if you are using mw docker you will always see some containers in the list, but Quabble will be at the top.

You can launch a shell either with the container ID or the name:

$ docker exec -it elegant_mclaren /bin/bash

The commands for executing the individual build steps can also be copied from the CI job console log output (in this case from the /workspace/src folder):

$ composer run --timeout=0 phpunit:entrypoint -- --testsuite extensions --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

You should now see the tests run in an environment that is more or less identical to the CI setup. If the command hangs, check to see if XDEBUG_TRIGGER is set or if your IDE has paused the execution at a breakpoint.

Interactively browsing the Quibble Mediawiki

For some purposes (e.g. manually running QUnit tests, or reproducing UX issues), you make want to interactively browse the MediaWiki that Quibble creates. To do this, simply export the browser port to your local machine by exposing the port in the docker invocation (add -p9413:9413). Now you should be able to reach the running MediaWiki at http://localhost:9413). If you added the matching Local Server to your PHPStorm setup and are using the modified docker image with XDebug, you should also be able to breakpoint requests in the usual way in the IDE.

External links

• Documentation