A test case management system (TCMS) can be used to organise numerous repetitive tests over several versions of a product release. It also provides simple interfaces to create, manage and execute tests for users with different permission levels. For the projects maintained by the WMF Language Engineering team, we are evaluating the use of Testlink as a test case management system. As a start we have included the tests for the Universal Language Selector that are manually run prior to MLEB releases. The following sections document the workflow used to create and execute the tests. Users with admin rights can create and manage tests. Users with tester roles can execute tests assigned to them.
More information about using Testlink can be found in the official User Guide
Users can be assigned different roles depending on the extent of their work inside the TCMS system. For the sake of simplicity, in the Testlink instance on language-tcms, we use 2 roles: admin and tester.
The admin users prepare the tests, plan them and assign them to testers
The testers can execute the tests assigned to them
Test assignments are done after test plans for the extensions/features have been drawn up
Before tests can be assigned, they have to be created and planned
Tests within the TCMS system are organised as Test Projects with Test Specifications. Test Specifications include Test Suites which are used to group the invidual Test cases.
Test Projects are created for individual product lines - for instance Universal Language Selector (ULS) . These can be created only by users with admin rights. While creating Test Projects a prefix can be specified that will be used in the unique IDs for all test-cases that are part of the project. An issue-tracker (like Bugzilla) can also be linked to a Test Project. The navigation path to create the Test Project is: From the User's Desktop view -> Click Test Project Management -> Click Create button
The next step is to specify the tests that will be part of the Test Project. For this the admin user has to click 'Test Specification' from the Desktop view and choose the Test Project for which the tests are to be created. In the panel on the left the Test Project name will be listed.
Click on the project name to view the Test Suite creation dialog. Click on 'Create' and add the name of the Suite. This should indicate the set of the test cases that will be found in the suite. It is recommended to create them as per a high-level scope of the features being tested. For instance: ULS-panel-interlanguage-logged-in. This would indicate that the tests are for the different features in the ULS panel that are accessible for logged-in users. A Test Suite can contain more Test Suites that can be used for further organising the tests. Once ready with a Test Suite, the test cases can be added in.
The Test Cases are the units that will be used to check specific functional aspects of the features. Each test case can have preconditions, several steps to replicate an action and expected result at each step. The testers will be able to verify the intended functions by following the steps and checking the expected results. These can be marked at the time of execution as PASS or FAIL. To add a Test Case, click on the Test Suite name (that can be seen after clicking Test Specification) under which the test is to be added. A dialog will be displayed on the right panel with options to add Test Suites and Test Cases
A new Test Case added
Click on the Create button under Test Case Operations. Add the title of the Test Case, a summary, preconditions (if any) and keywords (keywords can be added from the Desktop view and selected for Test Projects, Test Suites, and Test Cases). Click Create and the Test Case will be added in the tree in the left hand panel under the Test Suite. It will automatically be assigned a numerical id and the prefix (selected while creating the Test Project) will be added.
Adding a step in the Test Case
The next screen will display options to add Steps for the test and Expected Results.
A completed Test Case
A completed test case will look like this image. After creating test cases, the next step is to make them ready for execution
Test Plans can be created from the Test Plan Management link on the Desktop View. Click on the Create button to add a Test Plan. In the dialog displayed, the name, description of the plan can be added. It can also be marked as being Active and Public. After creating the Test Plan, the next step would be to add a Build and Test Set. The Build indicates the build release or version that is set to be tested. Unless specified the Test Plan cannot be marked for execution. This can be created by clicking on the 'Build / Releases' link from the Desktop View. For our example, we used - Build title: ULS MLEB 2014.3; Description: The build for the ULS tests for MLEB 2014.3 release.
Creating a Build to test
Next step would be to create the 'Test Set' i.e. the Test Suites and/or individual Test Cases that will be used to test the build that has been included in the Test Plan. Click on Add/Remove Test Cases from the Desktop view to see the dialog for adding test cases. Select the Test Plan from the left panel and click on a Test Suite (from the tree) that contains test cases. The Test Cases will be listed along with checkboxes that can be used to make selections.
Adding test cases to a Test Plan
Select the Build for tests and (optionally) the user you would like to assign the test to. The test assignment can also be done at a later stage.Select the test cases (all or individual) using the checkboxes and then click on Add Selected button. The view will change to reflect the selections made. Similarly test cases can also be removed. Repeat this process of selection for all the Test suites and/or Test cases as appropriate for the Test Plan.
Test Projects, Test Suites, Test Cases, and Test Plans can all be replicated from the TCMS interface at the time of creation. The forms will provide options from existing sets that can be chosen as the base. These can also be exported/imported using XML files.
New sets are necessary not just for creating newer versions of tests, but also for creating duplicate sets of Test Plans if more testers join in later
Test cases can also be assigned to users after they have been created. The users will have to be granted Tester roles by the Admin or Leader for tests to be assigned to them. Click on the Assign Test Case Execution link on the Desktop view. In the displayed dialog, select the Test Suite that contains the Test Cases to be assigned. Make the selections using the checkbox and dropdown next each Test Case and click 'Save'
Test cases assigned to users
The testers will now be able to see the tests assigned to them when they log-in
View presented to Testers. Tests can be executed from this screen
Log in using the user account credentials. Select the Test Project, Test Plan and then Click on Execute Tests from the Desktop View to start testing. All the assigned test cases will be displayed in the left panel. Select a test case and view the instructions. At the end select if the test has passed successfully (i.e all expectations have been met) or failed. Repeat for all the tests. Tests can also be skipped.
