Meetings/Test framework/2010-07-11

Test framework deployment Develop a Selenium test framework with samples and demos that can be run on the grid. Group: Release Engineering Start: 2011-07-01 End: 2013-12 Team members: Markus Glaser, Ryan Lane, Rob Lanphier, Mark A. Hershberger, Priyanka Dhanda, Dan Nessett

Test Framework Discussion Notes

Location: Wikimania 2010
Attendees: Ryan Lane, Markus Glaser, Benedikt Kämpgen, Siebrand Mazeland, Priyanka Dhanda

Short Term Goals

• Formalize the test framework roadmap. Make this visible to the community. (Robla and Priyanka)
• Documentation (Ryan Lane)
  • Document the Selenium Grid setup on a public wiki (both hardware and software).
  • Make all documentation visible to the openqa community for both feedback and as guidelines to others trying to do the same thing.
  • Make a visual representation of all test system interactions - Continuum, PHPUnit, Selenium Grid (Ryan Lane)
• Work in small iterative steps and make these steps visible.

For example: The first step should be clear documentation and a working example of how to write a test, run it against the grid and see the results.

• Define testing best practices.
• Encourage following code conventions and exporting tests to PHP.

We should encourage these best practices, but not frown upon tests that don't comply. There should be a process of sanitizing then and pulling them into the codebase.

• Investigate and integrate the Selenium Framework with Apache Continuum. Look into a code coverage tool.

Long Term Goals

• Find a good reporting strategy for test failures, code convention violations, etc.
• Figure out a process to let community members write IDE tests and then incorporate them into the code base.

Who should be is responsible for maintaining extension and core tests? Should it be the developer?

• Trigger a subset of tests on each commit and run all tests at some longer interval.
• Have requirements and tests closely tied together through the feature specification page.
• Make it easy for a bug submitter to submit test cases. Same with feature requests.

This way bug fixers and feature developers can use the test cases as a guideline and improve it as they go.

Outstanding issues with the current Selenium Framework

• Initial content on the wiki we're testing.

For now tests should be responsible for content generation and cleanup.

When the number of tests increase we can figure out a better way to bootstrap data.

• Add a configuration flag to allow tests to be run anonymously (Currently it requires login).
• Dynamically reconfigure the wiki being tested.
• Finalize the structure of the tests in the code base.

Now the test classes live in <ExtensionName>/tests/selenium. Each class is a suite with multiple test cases.

We should be able to run tests by a tag name and specify dependencies for each test suite.

• Test job queues.

This may be easier if we have a configuration to put a wiki in test mode. Ideally it should enable flushing queues through the framework.