Selenium/Ruby/Writing tests

Test automation provides channels for non-programmers and inexperienced programmers to be valuable contributors. Non-programmers, particularly people involved in the product and project side of software development, can contribute well designed acceptance tests for features. Less experienced people who know the Document Object Model and browser behavior can contribute page objects for the automated tests. And more experienced programmers can contribute test steps that tie together acceptance tests and page objects by automating browsers.

To get started with browser testing, see How to contribute.

Acceptance Test Driven Development

ATDD is a practice where we specify the behavior of software features in plain language sentences, often in some specific format. We then transform these specifications into executable code and threat them as automated acceptance tests for that feature. When the acceptance tests pass, the feature is known to be done. ("Done" has a specific meaning in agile software development: that the code is fit for use and ready to deploy.)

The canonical example of ATDD has the automated acceptance tests created before the code to satisfy them is created. In practice, this requires a level of cooperation between project/product people and development/test people that many organizations find difficult to achieve.

The current practice at WMF is to develop the feature specifications for Wikipedia software after or as the software is developed. There is nothing intrinsically wrong with this approach, although it is generally desirable to have testing activity match development activity as closely as possible, even to the point of moving testing activity *ahead* of development activity. Doing true ATDD is a great aid to focus and completeness of features.

In practice, acceptance tests for software features may be created and automated at any time, after which they serve as useful automated regression tests for those features. This is the point where non-programming members of the community, people with an interest in the proper function of Wikipedia software, can contribute.

We are using the Cucumber ATDD framework for browser automation. Cucumber requires that feature tests be specified as a series of statements of the form

Given <some initial condition>
When <I perform some action>
Then <I should observe some result> 

People from any area of the Wikipedia community are potential contributors of acceptance test criteria for features. Wikipedia editors would certainly like to be assured that their favorite tools in their favorite browsers work and continue to work in the ways that they expect; Wikipedia user advocates such as the Teahouse contributors and Ambassadors likewise. WMF staff are also invested in the behavior of software features, with perhaps the Product area being the most concerned. Finally, there is a community of people outside Wikipedia who are interested in browser testing in and of itself who are a significant potential source of automated acceptance tests.

Designing and defining excellent acceptance test criteria in the form of browser behavior can be tricky, and we aim to train our community to do it well. For more information, see Writing feature descriptions.

How to write WMF browser tests

Automated browser tests involve:

1. Features and Scenarios
2. Page Objects
3. Test steps
4. API-based setup methods

You'll also need to watch out for aspects of the test environment.

Potential contributors who are not programmers will be most interested in the description of Features and Scenarios. If you want to dig a bit deeper into how this project works, look at Page Objects. People actually creating end-to-end tests will be particularly interested in information about Test steps. All can read Quality Assurance/Browser testing/Very Basic Howto.

To run tests locally, see Running tests.

Tests are in a tests/browser subdirectory of each project, e.g.browse the code of the browser tests in core. The links section of the mediawiki-selenium README file lists the repositories that contain browser tests.

Features and Scenarios

Features and Scenarios in Cucumber are an implementation of a design pattern for Acceptance Test Driven Development, ATDD. They follow a generally accepted convention for such tests:

Feature: My Feature
  Scenario: Testing some aspect of My Feature
    Given <some initial condition>
    When <I take some action>
    Then <I should observe some result in the browser>

Any Given/When/Then statement may also have an arbitrary number of "And" clauses:

Scenario: Testing some complicated aspect of My Feature
  Given <some initial condition>
    And <some other condition>
  When <I take some action>
    And <I take some other action>
    And <I take yet another action>
  Then <I should observe some result in the browser>
    And <I should not observe some other result>

A Given statement should only ever mention starting conditions. A When statement should always be some sort of verb, or action. A Then statement always has the words "should" or "should not", or equivalent. While not required, it is good practice for technical reasons to use "should" when implementing the automated test steps later.

The Scenarios are the most important part of the tests. Take time to make them well considered, granular, and create as many individual Scenarios as necessary to adequately test the feature in question. A good question to ask is "when this test fails, what will that failure tell us about the software?" The language of the Scenarios is reported directly in the output when the test fails, so it is important that each step in each Scenario be designed well. This is the heart of ATDD.

Here are scenarios for a search test in use right now (from features/search.feature):

Feature: Search
  Scenario: Search suggestions
    Given I am at random page
    When I search for: main
    Then a list of suggested pages should appear
      And Main Page should be the first result
  Scenario: Fill in search term and click search
    Given I am at random page
    When I search for: ma
      And I click the Search button
    Then I should land on Search Results page

Feature files containing Scenarios are named "foo.feature" and reside in the /browsertests/features/ directory. Some good examples already exist.

Page Objects

Just as Cucumber implements an ATDD design pattern, we are using the "page-object" Ruby gem to implement a design pattern called Page Object. The idea of a Page Object is that every element on every page in the test suite is defined once and only once, in the context of the page being tested. This makes for ease of maintenance, since any change to that page element only requires updating one line in browser tests.

Such a design allows more radical changes as well. For example, it would be possible to swap out entire systems, say mobile for desktop, or a different language for English, by changing only the page objects and not the actual tests.

The definition of page objects for a simple search of Wikipedia looks like (from CirrusSearch browser tests):

class SearchPage
  include PageObject

  button(:search_button, id: "searchButton")
  text_field(:search_input, id: "searchInput")
  div(:search_results, class: "suggestions-results")
  ...
end

"SearchPage" is our class name for the page being tested. The code following "PageObject" identifies the elements that tests reference on this page – buttons, links, message boxes, etc. The syntax to identify an element is a w:Domain-specific language and is slightly odd:

element_type(:your_element_name, attribute identifying the element: "value of attribute", ... [more attributes] )

The first parameter to the function is your name for the element, such as search_button in the example above, prefixed with a colon. For readability and maintainability, we order the page elements alphabetically by this.

The elements documentation lists all the element_types available, there are elements corresponding to most HTML elements. For each element_type it presents the attributes you can to locate the element; these usually include class, css, id, index, xpath, and name for form elements.

A digression on Ruby: id: "firstHeading" is a key-value pair in a Ruby 1.9+ hash where the key is a Symbol object. Some online documentation uses the older Ruby 1.8 "hash rocket" syntax for this, with a leading colon, :id => "firstHeading". We prefer the former.

Locating by id is bulletproof, otherwise using a CSS selector can work well. Note that identifying by CSS does not use jQuery/Sizzle, instead it's the W3C CSS selector that the target browser implements. So you can't necessarily copy a jQuery $( 'selector magic'); for example you can locate with

 :button(:change_post_save, css: "ul.flow-edit-post-form .mw-ui-constructive")

but not

 :button(:wont_work, css: ".flow-edit-post-form [class^='flow']:last" )

You can usually specify more than one attribute to locate a single element (because we are using the watir-webdriver API for Selenium).

Often you need to locate a page element inside another, such as an item within a menu, or the title of the second item on the page :

• craft a CSS selector that does the right thing
• craft an xpath that navigates the DOM
• identify the parent and then nest the definition of the other element

Page-object files are named foo_page.rb and reside in the features/support/pages subdirectory.

Test steps

With a Scenario (or several) in place, and a Page Object defined for the page to be tested, now we can tie them together with test steps. This is the programming part, and it is helpful to have some grasp of the technologies being used.

An implementation of a Search test using the Page Object above looks like (from features/step_definitions/search_steps.rb):

Given /^I am at random page$/ do
  visit RandomPage
end
When /^I click the Search button$/ do
  on(SearchPage).search_button
end
When /^I search for: (.+)$/ do |search_term|
  on(SearchPage).search_input= search_term
end
Then /^a list of suggested pages should appear$/ do
  on(SearchPage).search_results_element.when_present.should exist
end
Then /^I should land on Search Results page$/ do
  @browser.url.should match '&title=Special%3ASearch$'
end
Then /^(.+) should be the first result$/ do |page_name|
  on(SearchPage).one_result.should == page_name
end

New 2014: Prefer expect( ) to trailing should exist

Each line in the .feature file becomes part of a regular expression in each test step. Just as we specify each element in each page only once and we specify each page only once, we also specify any particular aspect of the test such as text only once.

Since many tests may use a random page, RandomPage is specified separately. Because each line in the .feature file becomes a regular expression, we can using matching within the expression to specify strings to be searched for in the step, from the .feature file. We use a regex so we specify that text in the .feature file, and not in the steps file.

Note that the lines "Given I am at random page" and "When I search for:" are used twice in the .feature file. In the test steps, those lines are implemented once and called twice. Any single test step is implemented only once and may be used as often as necessary by the Feature. In the example above, we need only one test step to search for "main" and to search for "ma" using a regex as noted above.

In terms of architecture, it is Cucumber that ties together the lines in the .feature file and the test steps. It is the page_object gem that provides the means to identify elements on pages. The page_object gem is aware of both watir-webdriver API syntax and also pure selenium-webdriver syntax, so either approach is acceptable.

And for assertions, Cucumber uses the assertion framework called RSpec. This is why every "Then" statement in the test steps must involve expect() or (older), a "should" or "should_not" clause: otherwise the test will always pass and never fail. These are simple examples, but RSpec provides rich and complex ways to create potentially elaborate expectations for tests to satisfy beyond "should" and "should_not".

Update 2014: prefer expect( ... ) to trailing should should_not, see http://myronmars.to/n/dev-blog/2012/06/rspecs-new-expectation-syntax

Again, a Then step should always include an assertion of some kind, it shouldn't just end with e.g. when_not_present, which is a precondition for something else

Comprehensive documentation for Cucumber and RSpec is beyond the scope of this document, but is readily available.

Debugging

See Selenium/Ruby/Debugging

When tests fail

Is there a better page for this @skip? It is not a the same as a straight cucumber tag like @firefox

It's important that tests run by Jenkins be green. If a test is regularly failing that is tricky to fix, you can as a temporary measure skip running it.

1. create a Phabricator task in the code's project and in #qa to fix the problem.

1. add the @skip tag above the failing Feature or Scenario it
   • add a comment above the @skip tag indicating some actionable follow-up (TODO, FIXME), with link to the task, above the skipped test, so that this doesn't become a mechanism for simply silencing noise

Example:

# TODO: https://phabricator.wikimedia.org/T101190
@extension-geodata @skip
Scenario: Nearby link not present in main navigation menu

At the end of a run, cucumber will print "5 skipped".

As of May 2015 @skip is implemented in Gather and MobileFrontend. Note @skip is different from other cucumber tags such as @internet_explorer_10 or @smoke. They tag those tests you want to run when you execute cucumber -t @internet_explorer_10

If a test is so flaky that it needs to be skipped for the indefinite future, you might want to rethink the test implementation or simply remove the test. Testing is all about tradeoffs and, if you can't find the sweet spot in implementing a scenario, many times it may be better not to introduce it at all lest you degrade the trust in the test suite as a whole.

Skipping test programmatically

Internally @skip calls scenario.skip_invoke.

You can call this programmatically if you determine a test is inappropriate.

Another user of this is MediaWiki-Selenium, it supports @extension-name tags that skip a Feature or Scenario if extension name isn't available.

Miscellaneous

See also Quality Assurance/Browser testing/Guidelines and good practices

Timeouts

Tests take time to run, especially in a VM on SauceLabs across a network or on your own slow MediaWiki instance.

So tests may fail due to timeouts when waiting a few more seconds would allow them to complete successfully. Inserting explicit sleep NN calls in tests is bad design. Instead, poll for conditions:

• when_present() polls until an element can be engaged
• wait_until() polls until a condition returns true
• wait_while() polls until a condition returns false

All take an optional timeout parameter, the default is 5 seconds.

That aft5 test is a good example:

    page.wait_until(10) do
      page.text.include? 'Thanks!'
    end
    page.text.should include 'Your post can be viewed on this feedback page.'

This says: We'll hang out until the AFT post is processed. We know that the processing is finished when the page contains the text "Thanks". At that point we should have a message showing a link to the feedback page.

Parallel tests

If you run tests in parallel with parallel cucumber and phantomjs you may speed up test time dramatically. These steps got CirrusSearch tests running in parallel:

• Make sure all tests pass in PhantomJS. Mostly, this shouldn't require any work beyond installing PhantomJS, setting BROWSER to phantomjs, and running tests.
• Make sure your tests are in many small features rather than a few huge ones.
• Add gem "parallel_tests" to your gemfile and run bundle install.

Running tests is now a two-step thing. First run tests in parallel.

 bundle exec parallel_cucumber --nice -n 5 features/

Some of them will fail, because of general flakiness. You can rerun failing tests like this:

 cat cucumber_failures.log | xargs bundle exec cucumber

Common functions

We wrote the mediawiki_selenium RubyGem which provides several common functions such as logging in and creating a page. Quality Assurance/Browser testing/Shared features describes its features.

API-based test setup methods

To facilitate more deterministic browser tests, we wrote the mediawiki_api RubyGem. The API client can be used to create dependent wiki resources from within Given steps via the MediaWiki action API. See the RubyGem and API documentation for further info on each available method and required input parameters.

An API client is available from within your step definitions as api.

Given /^I create a new wiki article$/ do
  api.create_page "Test Article", "Test Article Content"
end

Example

Feature file:

Feature: Reference Drawer

  Scenario: Reference drawer appears when a reference is clicked
    Given I am on a page that has references
    When I click on a reference
    Then I see the reference drawer

Step definition:

Given /^I am on a page that has references$/ do
  api.create_page "Reference Drawer Test", "Test.<ref>Test reference</ref>\n<references />"
  visit(ArticlePage, using_params: { article_name: "Reference_Drawer_Test" })
end

API login

Many API actions require that the client has first logged in. This authentication will be done automatically given all the right environment variables have been set.

You can also explicitly login with the API as a particular user. For example, Echo notification browser tests use the client.log_in() API call prior to client.create_page().

Note that by design logging in with the API bypasses the browser, so you can't client.log_in() and then visit a page in the browser. (Although supporting this might be a possibility to speed up tests that log in, bug 71635.)

Anonymous tests

Browser tests should never execute an explicit logout, because it interferes with other tests using Selenium_user. Instead, browser tests should use the API to create the appropriate conditions before launching a browser session as an anonymous user.

Further information

This document does not describe every aspect of the design and architecture of the browser automation system. It is intended to explain enough that any interested person might get a good understanding of how things are put together and how to get started. The administrators of the project are happy to elaborate on aspects of the system not covered here.

Manual:Coding conventions/Selenium has the coding conventions we follow.