This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on the information here being up-to-date. See Selenium instead. |
- These notes date from January 2014. Since around July 2014 we no longer use Cloudbees, we have our own Jenkins Continuous Integration at https://integration.wikimedia.org/ci/view/BrowserTests/
This page discusses how WMF browser tests are run. For a detailed discussion of the architecture of Selenium and Webdriver itself, please see this chapter from the book The Architecture of Open Source Applications.
Selenium 2.0 is also known as Selenium Webdriver. For Webdriver, every browser that can be automated by Selenium contains an embedded HTTP server that accepts instructions as JSON data. The browser tests implement a Selenium client that communicates with the server embedded in the target browser.
Given the proper setup, it is possible (and usually desirable) to run the Selenium client code (the tests themselves) on an entirely different host than the Selenium server (the browsers being used for the tests). For example, consider the case of test code that runs on Linux or OSX in order to test in the Internet Explorer browser.
Running the tests is controlled by a Continuous Integration tool called Jenkins. Jenkins handles scheduling, reporting, and management aspects of essentially any process the user desires. In this case, Jenkins launches our test builds that run for each browser for each target environment. Jenkins launches each test with sufficient information for the test to connect to a service provided by Sauce Labs that launches a fresh virtual machine (VM) containing the appropriate browser for each test configuration. Each browser is listening for commands issued by the process running on the host where Jenkins is running. The Jenkins instance that WMF uses is supplied by a company called Cloudbees. Cloudbees has a partnership with Sauce Labs, and Cloudbees supports Open Source projects, so this is a convenient and economical solution for running automated browser tests.
The steps to run one build are:
- Jenkins timer goes off or a build is kicked off manually.
- Jenkins starts a "Build Executor" to monitor that particular build until it finishes.
- Jenkins Build Executor pulls code from git
- Jenkins Build Executor launches a Ruby process on the local host for each individual test.
- The Ruby process has sufficient information and credentials to issue a request to Sauce Labs to create and make available a VM with the desired browser listening for Selenium instructions.
- The test runs as the Ruby process on the Jenkins host communicates with the browser on the Sauce Labs host.
- When the tests finish, the Jenkins Build Executor collects the information from the test, saves the results according to instructions, issues any notices required, etc.
- The Build Executor within Jenkins ends and Jenkins itself starts a new Build Executor for the next build in the queue.
This arrangement has both advantages and disadvantages.
Advantages:
- No need for WMF to manage versions of Ruby and Ruby gems on the Jenkins host. This was the original reason for choosing Cloudbees.
- No need to manage the security for the Jenkins/Ruby host with regard to executing processes, opening ports, access over the internet, etc.
- Cloudbees is economical.
Disadvantages:
- Some configuration, for example, custom email templates for build notices, must be configured by Cloudbees staff. This can cause delays.
- Our economical plan with Cloudbees allows us only two Build Executors at any time.
- We would like to run tests in each build in parallel, but this has proven to be not possible with the current arrangement on the Cloudbees Jenkins host.
- Occasionally a build will fail due to a connection failure between the Ruby process and the Sauce VM. With no access to the host itself, we cannot diagnose these failures.
- Occasionally a build will fail because some process involved in launching and ending the Ruby client processes returns a non-zero result. With no access to the host itself, we cannot diagnose these failures.
In the long term we would like to run the automated browser tests from a Jenkins hosted by WMF. When we launched the project in 2012, we believed that this would not be possible. We now believe that it would be possible.
As a first step, we intend to run a suite of tests selected to be appropriate for a bare wiki with no test data against a new MediaWiki wiki created when the test is run. This first suite of tests will not communicate with Sauce Labs hosts, but instead will be run using a local headless browser called PhantomJS.
The first steps we put in place:
- Jenkins launches a process that creates a bare MediaWiki instance on the local host using a specific git branch.
- Jenkins stars a Build Executor.
- Jenkins pulls test code from git.
- Jenkins launches a PhantomJS process on the local host to use as a browser.
- Jenkins launches Ruby processes on the local host for tests that will use the local PhantomJS process pointed to the new local wiki.
- Jenkins reports the build status.
A number of other enhancements are coming to the browser test architecture:
- The ability to create test data such as pages and users on the target wiki at run time by using the MediaWiki API. This will enable better testing for local wikis and local development environments, rather than having to rely on the shared test environments at beta labs and test2wiki.
- The ability to launch browsers with particular custom settings. This supports projects like the Mobile "Nearby" feature.
- The ability to issue database commands directly to the local wiki. This will enable operations that are not possible through normal channels, such as fully deleting test data at the end of a test run.