Wikimedia Release Engineering Team/Project/End to End test language expansion

Goal

To continually increase adoption and lessen the maintenance burden of our shared testing infrastructure we plan to add JavaScript support to our end-to-end testing frameworks (in addition to Ruby) and simplify the current Ruby framework.

Dependencies

• Early adopters of the new JavaScript based framework

Milestones

1. Evaluate available JavaScript end-to-end testing frameworks.
2. Send out user satisfaction survey to get baseline data on use and issues.
3. Provide a pre-release JS framework to early adopter teams.
4. Release the full featured version
5. If successful, help other teams migrate old and/or create new tests with the new framework through workshops and education.
6. If successful, deprecate MW-Selenium and migrate remaining projects to new JS framework.
7. Send out follow up user satisfaction survey

1516Q4	1617Q1	Q2	Q3	Q4	1718Q1
Evaluate available JavaScript end-to-end testing frameworks. - task T108874 Prototype new JS framework. - task T131121 Send out user satisfaction survey. - task T131123	Major work is on hold/de-prioritized for Q1 (due to staffing restrictions).	Provide a pre-release JS framework to early adopter teams.	Release the full featured version	If successful, help other teams migrate old and/or create new tests with the new framework through workshops and education. Send out follow up user satisfaction survey	If successful, deprecate MW-Selenium and migrate remaining projects to new JS framework.

Impact

Movement

• Higher levels of test adoption by both WMF Engineering and our community of volunteer developers will improve the overall quality of new features going out to all of our users (third-party reusers of MediaWiki as well).

Foundation

• JS framework with fewer and less obscure dependencies could make end-to-end testing dramatically more accessible to new and experienced developers alike—many more of our developers understand JS toolchains than those for Ruby.

KPI

• ratio of # changes in test code by project owners vs. those made by members of Release Engineering (higher is better)
  • Over the fiscal year we should see an improvement in the ratio of changes in test code by project owners vs by members of RelEng over the baseline (TBD) (higher is better). Each quarter should show improvement as we implement the new features and increase adoption.
• User satisfaction of the new tool based on results of a user survey.
  • A non-trivial improvement in user satisfaction in the end-to-end test framework, as compared with the baseline (baseline survey to be sent in 16/17 Q4).