Wikimedia Engineering/WMF Tech Days 2012/Testing
WMF Tech Days Unconference, SF, 09/11/2012 Testing, Jenkins, Continuous integration
Antoine (hashar) Testing
Started writing tests in PHPUnit, but only ~10% and very few extensions. Need to write more easily, test-driven development.
- PHPUnit --> for PHP
- QUnit --> for Javascript
- TestSwarm --> scale out QUnit tests to multiple browsers
Maintained by jQuery team Inherently browser-dependent, so use
- TestSwarm by John Resig, rewritten by our Timo (Krinkle)
used by jQuery and WMF (and others)
Eventually 100+ virtual instances running various browsers >> This (or through SauceLabs) is not needed at all, this is all automated through BrowserStack, which has an API in addition to the GUI for in-browser usage by humans - this is already working for jQuery and for us in Labs. Going live really soon now.
use SauceLabs or someone for test resources, Selenium
Behavior Driven Development ? -> not that much. Chris has extensive knowledge about it.
Chris: ideally pyramid:
- thick set of unit tests at the bottom, one or more tests for every method
- mid-level exercise particular functions
- at the top small specific set of tests typically that's the BDD
Selenium, uses ruby language, rake (like make), rspec which is a BDD layer. Also "Cucumber" higher-level, "Water" wrapper...
Our PHPUnit tests are incomplete and poorly designed. We do not use mocking at all. For example, for database backed tests, we duplicate a database in a tempory base. That make lot of tests very slow. Similar issue with files where we have to create ton of files on disk.
Problem with mocking is some DB queries are extremely complex.
phpunit has "fixtures", lets you set up a DB for each test. Mockery is a simple mock system for PHP https://github.com/padraic/mockery
We have no code coverage, not even at line level, let alone branch testing.
NUnit creates tests by introspecting functions
Jenkins edit
snapshotting and testing every day, Jenkins runs phpunit (include parser) but nothing else. Antoine wants to get to be able to tag, test, give to Test Swarm to run qunit tests and release a version TestSwarm has to be able to report back failures to Jenkins, which can report to gerrit.
Also in core/tests :
- Selenium (not updated since January 2011), nobody runs them, would not pass :-(
- Jasmine in core/tests : a set of JavaScript tests in /wiki/tests/jasmine/SpecRunner.html , they run...
Even test documentation is weak, http://www.mediawiki.org/wiki/Testing_portal ?
Tests are coming for fundraising core.
Siebrand: Test-driven development requires tests written first, but then you have failing tests; tricky to check that in. > Timo: You write tests first and then the code, but both are comitted together, of course.
Ping pong coding: one write test, the other write the code. Switch every two hours.
So how to improve?
- measure test coverage and aim to improve
- people who have success with testing need to share on mailing lists
More visibility for/from Chris http://www.mediawiki.org/wiki/Page_Curation/Testing was a good test plan, but wasn't implemented
Chris involved a team of people to find out regressions between 1.19 and 1.20. test plan:
Some extensions have CI testing in Jenkins: https://integration.mediawiki.org/ci/view/Extensions/
Chris developed https://github.com/chrismcmahon/Page-Object-WMF-spike in Ruby. He has built abstractions for every item on the login page, e.g. https://github.com/chrismcmahon/Page-Object-WMF-spike/blob/master/lib/pages/login_page.rb and then has rspec file that describes a context that's behavior-driven development-like, e.g.
context "Navigate to UW should work after login"
Cucumber is a higher-level means of describing tests.
"Selka" probably coming in to work on testing.
Selenium is a reference implementation of Web Driever, a standard for sending test information to a browser. Facebook has a PHP implementation for/in Selenium...
how could we enable/encourage a developer reporting bug to create a test for it?
See also edit
Continuous integration main pages:
Some tools: