Selenium/PHP/Selenium Framework

SeleniumFramework provides a standardized way of writing Selenium tests for MediaWiki and MediaWiki extensions. The software is based on the PHPUnit unit testing framework and inherits much of PHPUnit's functionality.

Test framework deployment Develop a Selenium test framework with samples and demos that can be run on the grid. Group: Release Engineering Start: 2011-07-01 End: 2013-12 Team members: Markus Glaser, Ryan Lane, Rob Lanphier, Mark A. Hershberger, Priyanka Dhanda, Dan Nessett

Selenium's uses & flaws

Selenium is a tool for looking at an installation and trying to break it and filing bug reports, maybe in a mechanized way -- in other words, integration testing. But automated integration testing, as we've now experienced, frequently breaks; it's brittle, not robust. Selenium is an automated tests framework suited to integration testing: you start Selenium and it fires up a browser, performs scripted actions as a user would, and checks the actual output against the desired output. But that means that we have to define the desired output in a way Selenium can programmatically test, and so the tests break when we change things that actual users wouldn't care about.

For example, a test might define success as "this DIV should have the value 1". If we change the skin to output a SPAN element instead of a DIV element, then the test will fail. And even if a failed test is a legitimate signal of a problem, we have to start from scratch investigating what to fix and how.

We're thus phasing out the use of Selenium for MediaWiki -- see Test framework deployment#Status:_Permanent_hiatus. Sumanah 19:01, 1 July 2011 (UTC)

Installation

SeleniumFramework has the following major dependencies:

Selenium

SeleniumFramework curently uses Selenium Remote Control (Selenium RC) (and also should use Selenium Grid in the future.)

• Download Selenium RC from http://seleniumhq.org/download
  • Installation instructions for Selenium RC can be found at http://seleniumhq.org/docs/05_selenium_rc.html#installation

PHPunit

• Install PHPUnit via PHP's command line PEAR client

#  if needed, add the PEAR channels for PHPUnit and its dependencies
pear channel-discover pear.phpunit.de
pear channel-discover components.ez.no
pear channel-discover pear.symfony-project.com

# Install PHPUnit
pear install phpunit/PHPUnit

Testing_Selenium

Testing_Selenium is a PHP client for the Selenium Remote Control test tool.

• Install Testing_Selenium via PHP's command line PEAR client

# you can try "pear install Testing_Selenium", but it may give you an error if you don't specify the stable version
pear install Testing_Selenium-0.4.3

PHPUnit and Testing_Selenium must be accessible within your PHP path. If your PEAR installation is properly configured, then this should already be the case.

Configuration

Create a selenium_settings.ini file

The selenium_settings.ini file configures various settings for SeleniumFramework, such as the browsers that Selenium will attempt to use in testing and the base URL of the wiki to be tested.

selenium_settings.ini must be created in the root or your MediaWiki installation and should use one of the provided template files from tests/selenium. The template to use depends on the Selenium project (and, possibly, version of PHP) that you are running.

• If you are running Selenium RC with PHP 5.2, copy tests/selenium/selenium_settings_grid.ini.php52.sample into selenium_settings.ini.

Once you have created selenium_settings.ini, you will need to customize the settings it contains. At a minimum, you will need to set the wikiURL configuration value to match the base URL of the MediaWiki instance you are testing.

Update your LocalSettings.php file

Once the preceding steps are complete, you will need to update your LocalSettings.php file to enable SeleniumFramework.

To enable SeleniumFramework, you will need to add the following code to LocalSettings.php :

$wgEnableSelenium = true;
$wgSeleniumConfigFile = $IP . '/selenium_settings.ini';

• Hint: Depending on the folder where the tests are run, the selenium config file locator should point from the root.

Loading test configurations and suites

Once SeleniumFramework has been enabled, you'll need to tell it what tests to run.

To do this, you'll need to add a bit more code to LocalSettings.php .

The following examples show how to load various test suites.

# Basic example
require_once( "tests/selenium/suites/SimpleSeleniumConfig.php" );
$wgSeleniumTestConfigs['SimpleSeleniumTestSuite'] = 'SimpleSeleniumConfig::getSettings';

Here's a more complex example that loads the WikiEditor test suite. Note that WikiEditor must be installed for this to work.

require_once("extensions/WikiEditor/tests/selenium/WikiEditorSeleniumConfig.php");
$wgSeleniumTestConfigs['WikiEditorTestSuite'] = 'WikiEditorSeleniumConfig::getSettings';

Multiple MediaWiki instances

The configuration above assumes a single wiki for everything. It is possible to have two separate wikis (one driving the tests, and one being tested). More information can be found in Selenium Configuration.

Writing tests

Adding a test suite

To add a Selenium test, you must first create a test suite class that extends class SeleniumTestSuite. Look at SimpleSeleniumTestSuite.php for an example (which is in the tests/selenium/suites directory of the MediaWiki source.

Add test cases to the suite by implementing the addTests() function. If you have per-suite setup and teardown code, override the setUp() and tearDown() functions.

class MyExtensionTestSuite extends SeleniumTestSuite
{
 # Define only if you have per-suite setup code
 public function setUp(){
 	# Add setUp code here
 	parent::setUp();
 }
 public function addTests() {
 	$testFiles = array(
 		'extensions/MyExtension/tests/selenium/SimpleSeleniumTestCase.php',
 		'extensions/MyExtension/selenium/FooBarTestCase.php'
 	);
 	parent::addTestFiles( $testFiles );
 }
 # Define only if you have per-suite teardown code
 public function teardown(){
 	# Add setUp code here
 	parent::tearDown();
 }
}

Adding individual tests

In each test file listed in addTests(), create a class that extends SeleniumTestCase. Name the class after the file that contains the class, omitting the extension. For example: A file named SimpleSeleniumTestCase.php should contain class SimpleSeleniumTestCase.

For each test that you wish to run, add a public function to the class. Each function name should start with test. For example: testWidget().

Test functions should contain no more than one call to any of the various assert functions (like assertEmpty() or assertEquals()). The reason for this is that the logger only reports whether or not a test passed, not which assertions in the test passed or failed. If there are multiple assertions in a test function and the test fails, it will be impossible to know which assertion caused the test to fail.

If you have per-test setup and teardown code, override the setUp() and tearDown() functions.

setUp() and tearDown() are run for every test – not just at the beginning and end of all test runs.

class SimpleSeleniumTestCase extends SeleniumTestCase
{
 public function testBasic( ) {
 	$this->open( $this->getUrl() . '/index.php?title=Selenium&action=edit' );
 	$this->type( "wpTextbox1", "This is a basic test" );
 	$this->click( "wpPreview" );
 	$this->waitForPageToLoad( 10000 );
 	// check result
 	$source = $this->getText( "//div[@id='wikiPreview']/p" );
 	$correct = strstr( $source, "This is a basic test" );
 	$this->assertEquals( $correct, true );
 }
 public function testGlobalVariable1( ) {
 	$this->open( $this->getUrl() . '/index.php?&action=purge' );
 	$bodyClass = $this->getAttribute( "//body/@class" );
 	$this-> assertContains('skin-vector', $bodyClass, 'Vector skin not set');
 }
}

Examples

• For a very basic sample test suite and test case look in tests/selenium/SimpleSeleniumTestSuite.php and tests/selenium/SimpleSeleniumTestCase.php
  • You will need the following in the selenium_settings.ini file to run these tests

[SeleniumTests]
testSuite[SimpleSeleniumTestSuite] = "tests/selenium/SimpleSeleniumTestSuite.php"

• A working example can also be found at the PagedTiffHandler extension.

Test Wiki configuration

Multiple test suites that need different wiki configurations are run against a single test wiki.

Recording tests with Selenium IDE

You can record tests with Selenium IDE and still use them with the framework. In order to do so, you have to follow these steps:

• Record the test with Selenium IDE
• Export the test to PHP
• The IDE creates a method within the test class which has the name of the test. Copy the inner code of this method (not the function head) into the runTest method of SimpleSeleniumTestCase.
• Of course, rename SimpleSeleniumTestCase to a more descriptive name.

Running tests without Selenium Framework

In some cases, Selenium Framework will not be available on the wiki under test. This is especially the case when testing the installer. In this case, you need to tell the test not to use the dynamic reconfiguration and not to try to log into the wiki under test. This is done in the test suite like this:

class MyExtensionTestSuite extends SeleniumTestSuite
{
 public function setUp(){
 	# Add setUp code here
 	parent::setUp();
 	$this->setTriggerClientTestResources( false );
 	$this->setLoginBeforeTests( false );
 }
 ...
}

Running Tests

• To run tests, run "php tests/RunSeleniumTests.php" from the root directory.
• Old way (now, tests are in tests/...): To run tests, run "php maintenance/tests/RunSeleniumTests.php" from the root directory.

Requirements

• The test wiki should have a default configuration. This is the state that test suites should assume for a test wiki before setUp() is called.
• Each test suite needs to tell the test wiki what configuration it needs. This may be
  • files to be included for an extension
  • global configuration variables
• Could also be used to switch out the db and images directory.

(Do we need more fine grained control, like per test instead of per test suite?)

• At the end of a test suite, the wiki needs to go back to its original default configuration.

Missing

• Concurrent test suites point to the same wiki
• Whether this will work to change the db and images directory for the test wiki.
• If any extensions do custom configuration that is not an include or a global config variable.

Implementation details

Test suite:

• The test suite's setUp() makes a request to the test wiki with a request parameter whose value is the test suite name.
• After this the test can assume that the the test wiki is configured correctly for this test suite.
• The test suite's tearDown() sends another request parameter which will tell the test wiki to do back to using the default configuration.

Test wiki:
The following happens in WebStart.php only if a global variable $wgEnableSelenium is set.

• If request param setupTestSuite is set
  • set a cookie containing the test suite name
  • cookie is set to expire in 10 mins (Should this be configurable)

• If request param clearTestSuite is set
  • clear the cookie containing the test suite name

• If we find our testSuiteName cookie
  • Look for a global variable $wgSeleniumTestConfigs[testSuiteName]. If found, this is the test suite's function that is responsible for providing the following.

the test specific include files

the test specific globals (key value pairs)

• $wgSeleniumTestConfigs[testSuiteName] needs to be set in LocalSettings.php. The reason this is not a done via a hook is because this configuration needs to happen before Setup.php.

• In the case that it finds neither of the request params mentioned above nor the cookie, the Setup will proceed as usual with only the default configuration.

Test wiki configuration example:
As an example see extensions/WikiEditor/tests/selenium

• LocalSettings.php should contain

require_once("extensions/WikiEditor/tests/selenium/WikiEditorSeleniumConfig.php");
$wgSeleniumTestConfigs['WikiEditorTestSuite'] = 'WikiEditorSeleniumConfig::getSettings';

• WikiEditorSeleniumConfig::getSettings returns an array of include files and configuration variables needed by the extension.
• If a cookie has been set, indicating this is part of the WikiEditorTestSuite test run, we use the configuration we get fromWikiEditorSeleniumConfig::getSettings

Architecture

These files are part of the framework:

• tests/RunSeleniumTests.php includes test suites specified in extension directories.
• tests/Selenium.php provides access to selenium and implements common tasks.
• tests/selenium/SeleniumLoader.php loads all the necessary classes for the framework.
• tests/selenium/SeleniumTestSuite.php starts and stops selenium tests.
• tests/selenium/SeleniumTestCase.php provides some additional assertions.
• tests/selenium/SeleniumTestListener.php is the interface to result logging.
• tests/selenium/SeleniumConsoleLogger.php and tests/selenium/SeleniumHTMLLogger.php produce the actual output.
• SimpleSeleniumTestClass.php, SimpleSeleniumTestConfig.php and SimpleSeleniumTestSuite.php in the tests/selenium/suites directory provide examples.

The tests should be located here:

• Tests for core should be placed in the tests/selenium folder.
• Extensions should place their tests in extensions/EXTENSION/tests/selenium

Testing with a clean database and file state

testrunner                     wiki under test
----------                     ---------------
1.1 start selenium which in
turn starts a browser to
talk to the wiki under
test
1.2 send request for new test
with unique test id and tests
that will be fired         
                               2.1 create cookie with test id
                               2.2 create temporal resources 
                               according to tests list
                               2.3 create test tracker with timestamp
                               2.4 return success code
3.1 start testsuites via selenium
3.2 send a lot of individual
requests according to the tests
                               4.1 testrunner is identified by test id
                               4.2 reconfigure database and resources
                               according to test id
                               4.3 ? Do something with memcached ?
                               4.4 execute request
                               4.5 update timestamp in test tracker
5.1 send a teardown request
                               6.1 execute teardown, i.e. delete
                               all resources associated with 
                               test id
                               6.2 delete test tracker
                               6.3 return success code
7.1 stop selenium

Test style guide

This section focuses on general test style issues specific to Selenium Framework. For general issues of Selenium test style, see the test design considerations section of the Selenium documentation.

Leave no trace

• A test should leave at best no traces in the wiki.
• If this is not possible, the test should leave the wiki in a state which allows the test to be re-run.

Avoid brittle paths

Many developers will use the Selenium IDE to draft tests. This approach allows developers to rapidly rough out tests, but may generate locators that fail when used on other platforms or that are brittle and break when exposed to trivial edge cases.

When possible, avoid absolute paths. Instead, start with an element ID and work relative to the ID.

For example, if the Selenium IDE generated a path like: /html/body/div[2]/p[1] and div[2] has an ID of content, then use path //div[@id="content"]/p[1]

File naming conventions

The names for the test and configuration files should follow these conventions:

• All files should have a .php extension.
• File names for configuration files should end with Config. For example: SimpleSeleniumConfig.php
• File names for test suite files should end with TestSuite. For example: SimpleSeleniumTestSuite.php
• File names for test files should end with TestCase. For example: SimpleSeleniumTestCase.php

Keep log output in mind

Do not expect someone to read your test code. If a test fails, people will most likely read a log file. It is therefore very helpful to pay attention to descriptive log lines.

Do not log with echo

Often, the output of tests is processed further, e.g. by a continuous integration server. In this case, everything that is logged via echo will be lost. More particular, the framework is equipped to produce xml output compatible to phpunit, and echo messages are ignored in this file.

Test cases vs. suites

In order to get a differenciated logging, it is possible to use several test methods per test case, each of which produces an individual line in the log. Unfortunately, the setUp method is called for each of this methods, which makes perfect sense in phpunit but is not a behaviour we want with selenium. So instead, we use a method testSetUp() and testTearDown(), which are called only once per test case. Here is an outline of such an architecture:

class GeneralTestCase extends SeleniumTestCase { ...
   public function testSetUp() {
   ...
   }

   public function testFactboxExists() {
   ...
   }
   
   /**
   *@depends testFactboxExists 
   */
   public function testFactboxContainsRightStrength() {
   ...
   }
}

Here, you can even build in dependencies (see comment in the code above), i.e. if a test fails, succeeding tests will not be executed if they are dependent.

Number of assertions

Ideally, there is one assertion per test. However, this is often not manageable in reality. But it is possible to add a log line to an assertion that will be used in logs and xml output:

$this->assertTrue( $this->isElementPresent("body"), "Element body is  not present" );

Notes and further improvements

Add common test tasks

• Trigger user preferences
• Edit a page (initial support already implemented)
• Delete a page
• Upload images (already in PagedTiffHandler tests)
• Show edit and history mode
• Revert pages
• Refresh pages
• Cross Browser testing
• Stress / Load partial testing
• Please add to this list...

Add common assertions

• Text within an article
• Text in Headlines
• Text in Table of Contents
• Error messages
• External link
• Internal link
• Please add to this list
• Training Framework

Improvements

• Make test result logging more talkative
• Check prerequisites of MediaWiki configuration (initial support in PagedTiffHandler tests)
• Find a way to execute selenes recorded via Selenium IDE
• Refactor naming conventions
• Ways to configure the wiki for specific test suites or extensions.
• A way to bring the database to a clean initial state before each test
• Use the mediawiki extension to write collaborative Selenium tests

Further ideas

A test (or a single test suite) should be able to reconfigure the wiki according to its needs. Here, some serious security issues arise. This might be a possible way:

• Add a hook 'LocalSettingsEnd' at the end of LocalSettings.php
• Add some URL parameter which indicates which test is being run
• Within the extension, the hook code now changes the configuration

If the hook is not triggered in LocalSettings, no reconfiguration can take place.

Troubleshooting

Error about Testing_Selenium (0.4.3 is installed from pear along with PHPUnit) not being found

• Replace /PEAR/ by the actual path to the repository

General Tips

• Ignore port A proxy time you out port 4444.
• Uncomment any tests in RunSeleniumTests.php to run different test suites?

Backporting

Currently, the framework is only available in svn trunk and will not be available in a packaged version before MW 1.17. In order to make it run with older versions, you need to do this:

• get a fresh svn checkout
• copy tests/RunSeleniumTests.php and tests/selenium to your tests folder
• copy maintenance/Maintenance.php and maintenance/doMaintenance.php to your maintenance folder
• copy includes/SeleniumWebSettings.php to your includes folder
• edit SeleniumWebSettings.php and add these lines after LocalSettings.php is included (surrounding lines are given for context, in MW1.17 this starts at l. 113):

 # Include site settings. $IP may be changed (hopefully before the AutoLoader is invoked)
 require_once( MW_CONFIG_FILE );
}

if ( $wgEnableSelenium ) {
 require_once( "$IP/includes/SeleniumWebSettings.php" );
}

wfProfileOut( 'WebStart.php-conf' );

This is known to work at least with MW 1.15.3 onwards.

Experiences: Recent changes in handling of framework

• PEAR PHPUnit 3.4 and PEAR Testing_Selenium: Check whether properly installed, esp. PHPUnit/Framework.php
• Installation of selenium-rc is pretty easy:
  • Download selenium rc
  • start "java -jar selenium-server.jar"