Quality Assurance/Status 2012-2014

2012

2012-06-monthly

This month saw a big focus on hiring the QA Engineer and Volunteer QA Coordinator. We also continued to be focused on testing Article Feedback (including via an event on IRC with OpenHatch and new testing volunteers), and are working to get beta labs fit for use as a test environment for AFT and Editor Engagement (E2).

2012-07-monthly

Hiring a QA Engineer remains a high priority. Article Feedback Version 5 is now on 5% of Wikipedia, with a plan in place to increase that percentage over the summer. AFTv5 is being praised highly by the Wikipedia community, although a small number of power users are experiencing a particular problem. We isolated that problem and have a potential fix in place for deployment July 24. Work on the labs beta cluster continues, with AFT and TimedMediaHandler first priorities and the Editor Engagement project to follow. No community test events are planned right now, although the groundwork is in place for community test events related to Bugzilla tickets for Extensions and to the Visual Editor.

2012-08-24

This month saw an emphasis on hiring, with excellent candidates currently being considered for all the positions with close relation to QA. With AFTv5 in place in production, testing focus shifted to NewPagesFeed and Page Curation Toolbar. Due to conflicts of holidays, vacations, time of year, meetings, and general complications we decided not to hold an explicit community test event for NewPageFeed/Curation, but test environments and a test plan will be available for those interested to explore this new feature. Finally, an exciting announcement about the beta labs test environment is imminent, but not yet quite final.

2012-08-monthly

This month saw an emphasis on hiring, with excellent candidates soon to be hired for all the positions that will be closely related to QA. With AFTv5 in place in production, testing focus shifted to NewPagesFeed and Page Curation Toolbar. Due to conflicts of holidays, vacations, time of year, meetings, and general complications, we decided not to hold an explicit community test event for NewPagesFeed/Curation, but test environments and a test plan will be available for those interested to explore this new feature. NOTE: announcement for QA Engineer and possibly Mobile QA will have been made by the time this is published.

2012-09-06

Barring unforeseen circumstances and after examining many hundreds of applicants, hiring of final candidates for the open positions closely tied to QA (QA Engineer, Mobile QA Engineer, Volunteer Community Coordinator, Bug Wrangler) should all be complete within days. These new hires will greatly accelerate QA work in the near future. On the testing front, several key extensions are now being deployed automatically to the beta labs test environment, with AFTv5 to be the first key extension fully hosted on on beta in order to retire the prototype test host, which has become obsolete. This month also saw a renewed focus on browser test automation with the creation of an automated test for the UploadWizard extension being used by both QA and at least one WMF developer. This work will be extended and refined by the new QA Engineer in the very near future.

2012-09-monthly

Barring unforeseen circumstances and after examining many hundreds of applicants, hiring of final candidates for the open positions closely tied to QA (QA Engineer, Mobile QA Engineer, Volunteer Community Coordinator, Bug Wrangler) should all be complete within days. These new hires will greatly accelerate QA work in the near future. On the testing front, several key extensions are now being deployed automatically to the beta labs test environment, with AFTv5 to be the first key extension fully hosted on on beta in order to retire the prototype test host, which has become obsolete. This month also saw a renewed focus on browser test automation with the creation of an automated test for the UploadWizard extension being used by both QA and at least one WMF developer. This work will be extended and refined by the new QA Engineer in the very near future.

2012-10-monthly

Newly hired Michelle Glover and Željko Filipin will be testing software and working on browser-level test automation for both mobile and web platforms. Željko has particular expertise in automated testing and will be joining Antoine Musso and Timo Tijhof in the Netherlands for a "Continuous Integration Summit" in conjunction with Wikimedia Nederland Hackathon 2012.

2012-11-06

Grover was at the Mobile test summit, and has plunged into Perfecto device testing for mobile (a 3rd-party service that lets you test real devices over the internet). As of tomorrow, Filipin is going to NL to talk about CI, Labs, Jenkins, etc. In the meantime, Grover and McMahon are discussing making some mobile tests happen.

2012-11-13

Creating backlog of tests to be automated. Working with Quim and Michelle on potential first institutional community QA project.

2012-11-monthly

The team contributed to the community QA draft strategy and presented the Acceptance Test-Driven Development concept to Wikimedia Product/Project managers. Regression testing of software deployments is ongoing.

2012-12-18

Announcement of browser test status to wikitech-l successful. Proposals to form Features testing and Browser testing Mediawiki groups in place. Publicizing those. Regression testing of software deployments is ongoing, including first release of Echo to mediawiki.org.

2013

2013-01-22

Examining opportunities for community testing events, possibly monthly. Echo, AFTv5, and Visual Editor are all current candidates for testing, with VE likely for late January or early February.

2013-01-29

Public test event for VisualEditor is underway. At least one major issue identified causing data loss, and another issue identified with tool-assisted Chinese input, with more to come.

2013-01-monthly

We started to schedule opportunities for community testing events. Echo, AFTv5, and VisualEditor are all current candidates for testing. A week-long focus on VisualEditor's support for non-Latin characters uncovered at least one major issue causing data loss, and another one with tool-assisted Chinese input.

2013-02-05

Test event for Visual Editor concluded successfully, as we identified at least five important software issues and added at least one volunteer tester from the professional software testing community. However, we got fewer testers from the Wikimedia community than desired. Planning for upcoming test events is underway and we are working on improving the effectiveness of our Wikimedia community outreach.

2013-02-10

Test

2013-02-12

Low-key AFTv5 community test underway in support of new features before deploying to French and German Wikipedias

2013-02-25

Mobile file/image upload test in progress. Planning future events and coordination with E2 team and others in San Francisco.

2013-03-05

Concluded mobile app file upload test exercise. Preparing for new test exercise, possibly for Search. Exploring another collaboration with Weekend Testers.

2013-02-monthly

Concluded mobile app file upload test exercise. Preparing for new test exercise, possibly for Search. Exploring another collaboration with Weekend Testers.

2013-03-12

Preparing for a testing collaboration with Weekend Testers Americas to focus on E3 tools for new users.

2013-03-19

Introduced principals from WMF/WTA for Weekend Testing session Apr 6, follow up pending. AFTv5 release to enwiki postponed, dewiki, frwiki scheduled.

2013-03-26

Finalizing details for new-user features volunteer test event with Weekend Testers Apr 6. Tested AFTv5 release to German Wikipedia, French Wikipedia pending. Testing upcoming new features for Echo.

2013-03-monthly

Finalizing details for new-user features volunteer test event with Weekend Testers on April 6. Tested AFTv5 release to German Wikipedia, French Wikipedia pending. Testing upcoming new features for Echo.

2013-04-02

Weekend Testing on Saturday using Test Plan at http://www.mediawiki.org/wiki/QA/WTA_ACUX_Test_Plan_Apr_6. Echo testing underway.

2013-04-08

Testing Echo deployment to mw.o. Processing ACUX input from Weekend Testing event Apr 6

2013-04-16

More Echo testing. Investigating issue with IE and ResourceLoader and style sheets

2013-04-monthly

We collaborated with Weekend Testing Americas to investigate new Account Creation UX features with the E3 team, and tested Echo deployments with the E2 team. We are investigating an intermittent failure with UploadWizard for Firefox, and a styling issue with ResourceLoader in IE.

2013-05-21

Tested Visual Editor at Test Summit peer conference, several issues reported. Tested Echo deployments. Upcoming work with Flow, more with GuidedTours, more with ULS.

2013-05-monthly

In May, QA worked with a number of parties both in and outside the Wikimedia Foundation to test Echo, VisualEditor, Universal Language Selector, and other projects. We began an official QA mailing list.

2013-06-03

QA is bringing ULS tests online. We made a test spike for VisualEditor and will expand that with our new OPW intern. Still working to turn builds from red to green: tweaking tests, shepherding bugzilla tickets for test environments, etc.

2013-06-monthly

This month saw a QA focus on automated browser tests. Besides creating new tests and new builds, and reporting issues identified by tests, we conducted a training session in San Francisco to create automated tests for the Wikilove feature. We continue to support all WMF software development projects, with the VisualEditor being a particular focus in June.

2013-07-monthly

This month QA made contributions to the VisualEditor, UniversalLanguageSelector and Mobile web projects, among others, finding and reporting issues in a timely manner. Our intern with the Outreach Program for Women is working on more automated browser tests. We continue to engage our community on the QA mail list and in live sessions, where we have several contributors (see Volunteer coordination and outreach).

2013-08-monthly

This month QA began collaborating closely with Release Engineering to coordinate improvement of reporting, monitoring, and testing software releases. Our goal is to make our frequent software releases even more reliable than they already are, and to use the tools and systems in place today such as the beta labs cluster to make those reliable releases even more frequent.

2013-09-monthly

This month, we wrapped up Rachel Thomas' Outreach Program for Women internship successfully. Rachel helped us extend our browser test coverage of VisualEditor. Besides our ongoing collaboration with Wikimedia Foundation development projects, we are also engaging the greater community on the QA mailing list, where we discuss both code contributions and general QA topics.

2013-10-monthly

In October the Quality Assurance project brought on a new software tester for VisualEditor, Rummana Yasmeen, who became productive quickly, reporting a number of issues and validating a VE deployment. QA spent a significant amount of time with the WMF Language team, using automated browser tests to find issues with the UniversalLanguageSelector and other Language software. QA continues to expand the utility of the beta labs test environment, supporting the new Flow extension there, and increasing the role of beta for deployments. Membership in the QA mailing list continues to grow, and October saw more contributions to the automated tests from volunteers. QA will participate in the Google Code In project as well.

2013-11-monthly

November saw significant improvements to the QA documentation on mediawiki.org contributed by both staff and volunteers. Participants in the Google Code-in program made even more contributions, to both documentation and browser test code. The QA team welcomed new staff members Rummana Yasmeen and Jeff Hall, who made immediate contributions to the VisualEditor project and to the browser test automation.

2013-12-monthly

In December, the Quality Assurance team worked particularly closely with the Mobile team, both supporting automated testing and also helping fix issues with Beta labs and with Jenkins. We continued to work with the teams from Language engineering, VisualEditor, Flow, Multimedia, Wikidata, and Search, as well as participated in the Google Code-In event. We are in the process of creating new support not only for automated browser testing, but also for API testing, test data creation, and monitoring of both test and production environments.

2014

2014-01-monthly

January saw the QA team working closely with the Mobile team in particular to enhance the existing suite of test for MobileFrontend. We also participated in the discussion of the Release Engineering deployment process at the architecture summit. Hiring is underway for two open positions, QA Automation Engineer and Test Infrastructure Engineer.

2014-02-monthly

In February, we updated our 3rd-party Jenkins instance to use Jenkins job builder configuration rather than Jenkins templates. Now our 3rd-party Jenkins builds matches the WMF Jenkins build scheme, giving us maximum flexibility for when and how these jobs are run in the future. Also, we laid the groundwork for several significant new test features to be announced in the near future.

2014-03-monthly

The QA team continues to identify and report issues in a timely way. Of particular interest in March was that an automated test uncovered an issue in the interaction of the MobileFrontend and VisualEditor extensions. This is exactly the kind of cross-cutting concern that our QA systems are designed to uncover. It is likely that we will be in a position to discuss these systems at the Wikimania conference in London.

2014-04-monthly

This month saw the QA team working closely with the MobileFrontend team to extend and refactor their test suite. We also made great progress in running many of the browser test suites on headless Firefox instances in builds controlled by WMF Jenkins. Work on the WMF Jenkins browser test builds will continue in order to take advantage of the power and flexibility we have there.

2014-05-monthly

The expertise of our new Automation Engineer, Dan Duvall, will allow us to update, maintain and modernize our development environments using Vagrant virtual machines, keep our Puppet site configuration running properly, and contribute to the whole delivery pipeline as we continue to improve our ability to deploy software features to Wikimedia sites quickly and safely.

2014-06-monthly

This month saw significant improvements to the MediaWiki-Vagrant development environments from new WMF staff member Dan Duvall. We have completed support for running the full suite of browser tests on a Vagrant instance under the VisualEditor role. In the near future, we will extend that support to the MobileFrontend and Flow Vagrant roles, as well as making general improvements to Vagrant overall. Another great QA project is from Google Summer of Code intern Vikas Yaligar, who is using the browser test framework to automate taking screen captures of aspects of VisualEditor (or any other feature) in many different languages, for the purpose of documentation and translation.

2014-07-monthly

This month, the QA team finished two significant achievements: after porting all the remaining browser tests from the browsertests repository to the repositories of the extensions being tested in June, as well as porting a significant set of tests to MediaWiki core itself, we completely retired the Jenkins instance running on a third-party host in favor of running test builds from the Wikimedia Jenkins instance, and we deleted the /qa/browsertests code repository. These moves are the result of more than two years of work. In addition, we have added more functions to the API wrapper used by browser tests, improved support for testing in Vagrant virtual machines, added new Jenkins builds for extensions, and improved the function of the beta labs test environments by preventing database locks and stopping users from being logged out by accident.

2014-08-monthly

Having completed the migration of our Continuous Integration infrastructure from a third party host to Wikimedia's own Jenkins instance, we are thinking about improvements and changes for future work. We aim to improve performance for Jenkins and also for beta labs. We are looking into creating other shared test environments along with beta labs to better support changes like we did this month with HHVM and with a security and performance test project. We also continue to improve the development experience with Vagrant and other virtual machine technologies.

2014-10-monthly

This month we welcomed Elena Tonkovidova to the WMF staff as our new QA Tester. Elena will be working closely with Rummana Yasmeen in that role. In support of our QA efforts, we invited Elisabeth Hendrickson, author of the book "Explore It!", to a well-attended meetup at the WMF office for a presentation about Exploratory Testing. Other projects for the month include adding a Ruby style checker Rubocop to our Jenkins builds, creating a Ruby style guide, further enhancements to the Vagrant development environments, updates to the beta labs test environment, and other projects.

2014-11-monthly

In November we made significant improvements to the Vagrant development environments, and also conducted a poll of Vagrant users to guide future development. We sorted out issues with HHVM on beta labs, and we made improvements to Jenkins and Zuul for speed and efficency. We introduced the Ruby linter "rubocop" to all of the Jenkins builds, one more step in managing technical debt and improving the quality of our browser test code.