Selenium/Ruby/Browser testing

Browser testing Automated testing of MediaWiki within various browsers to improve quality Group: Wikimedia Release Engineering Team Start: 2012-01-10 End: 2014-11 Team members: Željko Filipin Lead: Dan Duvall

Rationale

Why automate tests at the browser level? They find bugs!

Browser tests are most useful for testing:

• JavaScript: different browsers interpret JS in different ways.
• Features requiring navigation: unit tests don't usually navigate.
• Fragile Features: if something breaks often, we want to test it regularly.
• Critical Features: features that must work all the time.
• Beta features: checking that they work in all browsers.

It's tedious to manually test Wikimedia sites using the multiple browser-operating system combinations that are out there, so we do automated testing to find those errors.

• We run periodic browser tests via Jenkins on SauceLabs to test the latest master code on the beta cluster in multiple browsers
• Some projects run browser tests on each check-in to gerrit as part of Continuous integration, see Continuous integration/Browser tests.

We use Cucumber because it lets you write tests in plain English; you don't need to know much Ruby to be effective with Cucumber. Cucumber implements an idea called Acceptance Test Driven Development. The plain English test specifications open a communication channel with non-technical people who wish to contribute to browser test automation.

How to contribute

Investigate a broken issue

• Analyze failing tests in Jenkins browser test runs.
• Investigate a recent test that's failing
  • Drop in to #wikimedia-releng ^connect and ask if it's a known issue.
  • Look at the test's Build Artifacts. There may be .png files that are screenshots of failed tests.
  • Watch the screencast of the failing test and/or view the test's code.
• If you manually go through the test's steps on a wiki site (see Quality Assurance/Feature testing) and the test failure reveals a problem, then file a bug against the feature under test, mentioning the browser test that exposed the bug.

This doesn't require any local code at all, it's all on the web.

Fixing a broken issue

The next level is to fix broken tests and write new ones.

• Follow the browser testing setup instructions
• Run browser tests yourself, pointing to some MediaWiki install
• If a test fails but the feature under test seems to be working, see if you can figure out what's wrong with the test. Often it's a change in the text or CSS that the test looks for. You can try to fix the test locally, and rerun it.
  • If you find an error in a test, file a bug.
  • Going further, contribute a fix for it in gerrit.
• You can improve browser tests and write new ones. We use a high-level language that makes tests more approachable.

None of these steps require you to have a working local MediaWiki installation, though you may find it easier to deploy a MediaWiki-Vagrant virtual machine "appliance" which automates installation of some of the tools you need and obtains the MediaWiki source code with git. This also gives you a local working MediaWiki instance, so you can go a level deeper and modify the code of the feature under test to fix it yourself.

Resources

• Current test results.
• We use given-when-then testing format.

External links

• Cross-browser testing is a resource often use for specific testing