Language Testing Plan/LanguageTestingWorkflow

THIS IS A PROPOSAL

The Wikimedia Language Engineering team presently uses the Cucumber and Selenium to test implemented features in web-browsers. Besides these automated tests implemented using Ruby, the team also tests manually when automated tests are not feasible or not implemented. For all tests the scenarios to be tested are written in the Given-When-Then format. This can be explained as:

Given a Situation
When a certain event occurs
Then a certain result is expected

This document outlines the process and best practices to implement tests that fully cover the user interface functionalities for repeated runs.

Phase 0: Conceptualization

When a feature is identified for implementation, the details of its functionalities, interface design and interactions are ascertained by the Product Owner in consultation with other stakeholders and described in the design documents. These descriptions are the blueprints for the developers when they implement the functionalities through code. It is imperative that the functionalities are clearly described for all possible user activity. A means to this end is the conceptualization of the individual Given-When-Then scenarios, that provide the developers with the description of the events that can be executed under different conditions and the intended reactions.

Phase 1: Scenario Building

The Given-When-Then scenarios are the smallest levels of activities that can be put together to describe a functionality. It is if extreme important to create the scenarios from the defined designs, including user interface mockups. This ought to be done by simulating interactions by the user on the dialogs, forms, buttons, input boxes and other user interaction elements described in the mock-up.

The number of scenarios can be extremely elaborate at this stage. It is also preferable that the original concept creators are not consulted at this point, to avoid leading the scenario writer.

Completed: A list of Given-When-Then scenarios of possible user activities for all functionalities is prepared
People Involved: Language Engineering QA

Phase 2: Feature Review and Consensus

The list of scenarios created in Phase I is the foundation for all tests that need to be run against the feature. This exercise of granularity may also identify any missing functionalities or scenarios that may have been overlooked during the original design decisions. At this stage it is important to verify these inconsistencies with the Product Owner and/or UI Designer. This may or may not require the scenarios to be rewritten or modified.

In cases where the scenarios are written for extension of already implemented features, the original developers' may also be consulted.

Completed: The GWT scenario list is ratified for consistency with the intended feature
People Involved: Language Engineering QA, PO, UI Designer, (Developers)

Phase 3: GWT Scenario Refinement

In the next stage the ratified scenarios need to be refined for actual use. This will have to be done depending upon how the scenarios will be used. For automated tests, the GWT scenario descriptions will have to be reformatted and committed into the appropriate gerrit repository as .feature files, suitable for writing the step definitions. Since this step may somewhat reduce the verbosity of the scenario, the original set of scenarios should remain unchanged to aid the manual tests. The text for the .feature files should be checked against the original GWTs for any loss of information.

Completed: The GWT scenarios identified for the automated tests are refined and submitted to the repository as .feature files
People Involved: Language Engineering QA (Scenario writer and code writer)

Phase 4: Implementation

The test scenarios are now ready for implementation.

Automated

The process for writing the automated tests is elaborately described in the QA document.

Completed: The code for the scenarios in the .feature files is written and set in the build-system
People Involved: Language Engineering QA (Code writer) and WMF QA team member (pairing)

Manual

The tests that are not set for automated tests are written as manual test cases, with 'execution steps'.

Given I am in the Main Page of the wiki
When I press the 'Log-In' link
Then I should see the 'Log-In' dialog

Test Case 1
Step 1 : Go to URL-of-Main-Page-of-the-wiki
Step 2 : Click on the 'Log-In link on this page

Are you taken to the 'Log-In' page?  - Yes/No

Completed: The scenarios for the manual tests are written as manual test cases i.e. it contains the steps for executing the test and for reporting the result
People Involved: Language Engineering QA (Scenario writer)

Phase 5: Result Collection and Analysis

Manual

(WIP) The results of the tests are collected from the TCMS. Failed tests are verified and bugs filed.

Completed: Regressions, bugs etc. are verified and submitted in the bug-tracking system for prioritization
People Involved: Language Engineering QA