We use manual testing as well as automated testing to QA our products.
Our manual testing frequently occurs as part of our sprint process. After code has gone through engineers and design we hand it over for QA.
Manual handover
editFirst it is important to check we have instructions on how and where to test the change in the description, given we have many test environments identifying where testing can take place.
We have 2 environmentsː
- The beta cluster, which updates frequently as code is merged
- reading-web-staging.wmflabs.org, which is specialized for testing article content
Depending on what is being tested usually one is more suitable than the other. It's extremely important to clarify where to test to avoid testing false positives.
staging
editStaging can be updated by ssh'ing into the machine (readingwebstaging.eqiad.wmflabs). Always run sudo su mwvagrant
before doing anything else so that the logs of what you are doing can be seen by other users.
Updates usually involve either of these commands:
cd /srv/mediawiki-vagrant/ vagrant git-update git fetch && git checkout origin/master
If this doesn't work, it may be because a change has been cherry picked (see below). In this case, you should either rebase the cherry pick onto master or revert back to master if the change is no longer needed.
cherry picking changes
editSometimes for risky code we may cherry pick changes to staging. Right now we don't have a good process for managing this kind of change other than commenting on the tickets that need these changes.
Changes are last in first out so there is always a risk staging is left in an unexpected state. It is up to the tech-lead to keep track of these changes.
When cherry picking changes, always refer to the "design sign off" and "QA" columns of the kanban board and when done leave a comment.
LocalSettings
editIt may also be necessary to update LocalSettings.php (either to disable/enable the content provider) or to turn on new feature flags.
On staging, there is a file /vagrant/mediawiki/Staging.php where changes should be added to.
When making a breaking change (e.g. switching language) we should mail the team mailing list. Recently, I've been exploring making use of wgSiteNotice to inform users of the wiki without need for email. You'll find some commented out lines down the bottom of the Staging.php file.
Automated testing
editBrowser tests
edit- What are browser tests? Quality_Assurance/Browser_testing
Node.js
editWhat are browser tests for?
editA browser test should ideally be written to test behaviour. Wherever possible, favor unit tests. Usually a browser test is warranted in one of the following scenarios"
- You are testing behaviour of a workflow dependent on one or more events e.g. "When I click the browser back button the overlay closes"
- The thing being tested is dependent on configuration either on user or site level e.g. "When I have disabled page previews I do not see page previews"
- The behaviour of the thing being tested is subject to change by another extension "e.g. "When VisualEditor is installed, the default editor is the wikitext editor."
Running browser tests on the beta cluster
editSet the following variables, using an existing account for username and password:
export MEDIAWIKI_URL=http://en.m.wikipedia.beta.wmflabs.org/wiki/
export MEDIAWIKI_API_URL=http://en.wikipedia.beta.wmflabs.org/w/api.php
export MEDIAWIKI_ENVIRONMENT=beta
export MEDIAWIKI_USER=YourUsername
export MEDIAWIKI_PASSWORD=***
Note that certain tests may require your username to have certain permissions set.
Simulating browser test run on an unmerged patchset.
editThe way we test such change is to create a copy of the existing Jenkins job, disable all notifications and then fetch and checkout the patchset.
Steps:
- For example, you want to test 298959
- Log in to Jenkins
- Click New Item link in the left sidebar
- A form would show
- Item Name: selenium-(repository)-(gerrit-patch-number), so in this example selenium-Wikidata-298959
- Copy existing Item: selenium-(repository), so in this example selenium-Wikidata
- Save form
- Go to the job configuration
- Click Source Code Management > Git > Advanced
- Change Refspec from +refs/heads/*:refs/remotes/origin/* to refs/changes/59/298959/1
- You can get refs/changes... from Gerrit patch
- Go to the patch 298959
- click Download
- URL in Pull will be git pull https://zfilipin@gerrit.wikimedia.org/r/wikidata/browsertests refs/changes/59/298959/1
- You only need the last section, refs/changes/59/298959/1
- You can get refs/changes... from Gerrit patch
- Change Branch Specifier (blank for 'any') from master to FETCH_HEAD
- Change Refspec from +refs/heads/*:refs/remotes/origin/* to refs/changes/59/298959/1
- Uncheck Build periodically
- Delete the sections:
- Execute shell (only one that starts with # Attempt to figure out MediaWiki branch being used and fetch it out)
- Editable Email Notification
- IRC Notification
- Click Source Code Management > Git > Advanced
- Save and trigger a build of that item-name to run browser tests.
Troubleshooting "flakey tests"
editSometimes our beta cluster integration jobs fail, even though they pass when we merge code and a manual test shows that functionality is working as expected. This is often due to problems with the beta cluster's availability or performance and how the tests are written not the thing they are testing.
Use you can point Selenium jobs to unmerged patches, to test changes to browser tests by cloning an existing job by clicking "New Item" on the dashboard and using "copy from". This results in a duplicate job with a different name e.g. https://integration.wikimedia.org/ci/view/Reading-Web/job/selenium-MinervaNeue-BugFix/
You can use this job to point run tests from another code path:
- Click configure
- Find on page "gerrit". In build "Execute shell" you'll see an explicit checkout command. You can update this line to check out any code for that repository.
- Run the build (using "Build now")