Wikimedia Release Engineering Team/Project/End to End test language expansion
Goal edit
To continually increase adoption and lessen the maintenance burden of our shared testing infrastructure we plan to add JavaScript support to our end-to-end testing frameworks (in addition to Ruby) and simplify the current Ruby framework.
Dependencies edit
- Early adopters of the new JavaScript based framework
Milestones edit
- Evaluate available JavaScript end-to-end testing frameworks.
- Send out user satisfaction survey to get baseline data on use and issues.
- Provide a pre-release JS framework to early adopter teams.
- Release the full featured version
- If successful, help other teams migrate old and/or create new tests with the new framework through workshops and education.
- If successful, deprecate MW-Selenium and migrate remaining projects to new JS framework.
- Send out follow up user satisfaction survey
| 1516Q4 | 1617Q1 | Q2 | Q3 | Q4 | 1718Q1 |
|---|---|---|---|---|---|
|
Major work is on hold/de-prioritized for Q1 (due to staffing restrictions). | Provide a pre-release JS framework to early adopter teams. | Release the full featured version |
|
If successful, deprecate MW-Selenium and migrate remaining projects to new JS framework. |
Impact edit
Movement edit
- Higher levels of test adoption by both WMF Engineering and our community of volunteer developers will improve the overall quality of new features going out to all of our users (third-party reusers of MediaWiki as well).
Foundation edit
- JS framework with fewer and less obscure dependencies could make end-to-end testing dramatically more accessible to new and experienced developers alike—many more of our developers understand JS toolchains than those for Ruby.
KPI edit
- ratio of # changes in test code by project owners vs. those made by members of Release Engineering (higher is better)
- Over the fiscal year we should see an improvement in the ratio of changes in test code by project owners vs by members of RelEng over the baseline (TBD) (higher is better). Each quarter should show improvement as we implement the new features and increase adoption.
- User satisfaction of the new tool based on results of a user survey.
- A non-trivial improvement in user satisfaction in the end-to-end test framework, as compared with the baseline (baseline survey to be sent in 16/17 Q4).