Wikipedia.org Portal A/B testing/dev process

Portal A/B Testing Dev Workflow

In progress work

While work is being done on a feature, it should be posted to a Wikimedia ‘people’ page for team members to see progress. Each developer is given an account on people.wikimedia.org, and the work should be placed in an ‘in-progress’ folder in the users account.

Eg: https://people.wikimedia.org/~jdrewniak/in-progress/index.html

The contents at this URL should change frequently as a feature is being produced. It should be used to show in-progress work to any interested parties. Currently, the contents at this URL must be updated manually.

The ‘in-progress’ page should at the very least be updated when a patch is committed to code review.

In Beta

Once a patch passes code-review, it is automatically merged into the Portal Repo. At this point it is in theory ready for production. Before deploying to production however, the patch can be tested on the beta site. The beta site automatically updated to the latest commit in the repo about every 15 min or so.

The beta site is available here: http://www.wikipedia.beta.wmflabs.org/

Beta should be used to catch errors before they go to production and get ‘signed-off’ on a feature before it is deployed to production.

Portal Version Tool

Sometimes the beta site takes a while to update itself. To check which commit the site is running, visit the portal version tool:

http://portalversion.j4n.co/

This portal version tool finds the current commit that the beta page and the production page are running.

Once you know what commit the page is running, you can reference that commit ID with the repo history here:

https://phabricator.wikimedia.org/diffusion/WPOR/history/master/

Eg: If the portal version tool show the beta page is on ad2e8d9815a1, you can look at the repo history and see if ad2e8d9815a1 is the latest commit in the repo or not. If it’s not, then you know the beta site hasn’t updated itself yet. This tool can also be used to check if a deployment was successful or not. If you're expecting the production page to run a certain commit after it's been deployed, and the version tool shows it's not, then their could have been a problem during deployment.

To make it easy to cross-reference the commits to the repo with phabricator tasks, the commits should be prefixed with the phabricator ticket number.

Deployments

The portal page is deployed through the SWAT schedule:

https://wikitech.wikimedia.org/wiki/Deployments

For historical purposes, and for seeing what was deployed when, you can view the mediawik-config repo history here:

https://phabricator.wikimedia.org/diffusion/OMWC/history/master

This repo history is long, any many projects are deployed through this repo. To spot which commit was a portal deploy, look for the name of the portal developer near the commit.

The commits to this repo should also reference the last phabricator ticket this commit addresses.

Eg: Looking at the mediawiki-config repo and portal repo side by side, we can see that the ticket T132520 was merged into the portal repo on May 6th, and then deployed to production (throught the mediawiki-config repo) on May 10th. Portal repo on the right, mediawik-config repo on the left.

Documentation

Once a test has been completed, the A/B testing documentation should be updated here:

https://www.mediawiki.org/wiki/Wikipedia.org_Portal_A/B_testing

The timeline section should be updated. Figuring out when the test ran can be inferred from the SWAT repo history mentioned above.  When the analytics results are available, a ‘results’ section should also be added to the test, with a link to the analytics report, or a summary of the finding if no report was created.