Talk:Requests for comment/Unit testing

About this board

P858snake (talkcontribs)

PHP is pretty much static/similar on all platforms, so running PHPUnit in CruiseControl or on your localhost is enough to know that the code is good (if it passes the test). -- This statement is mostly only of theoretical value, because recently we have seen some tests that were/are still broken on CC but not locally, or that were broken locally but not on CC.

This post was posted by Peachey88, but signed as

πŸ˜‚ (talkcontribs)

Yes this is a problem. Some of it's differences in setup between ci.tesla and people's local installs. Differences in pass/fail between setups is usually an indication of a problem that needs fixing anywayΒ :)

Krinkle (talkcontribs)

May also be related to CruiseControl install using SQLite and (most local installs) using MySQL.

πŸ˜‚ (talkcontribs)

That's part of the reason. It's why I always test phpunit on bothΒ :)

Hashar (talkcontribs)

Maybe we could add CruiseControl build against postgre and MySQL and just running the Database group.

πŸ˜‚ (talkcontribs)

Can we please not install pgsql on ci.tesla? The reason we use sqlite over mysql is because it's nice and lightweight and we're on a VM.

Bryan (talkcontribs)

Also, Windows. Some tests fail on Windows that do not fail on Linux.

πŸ˜‚ (talkcontribs)

I'm not looking to build a test cluster with multiple VMs running different OSs.

Reply to "PHP similarity"

additional wikitech-l discussion re testing interactions

Sumanah (talkcontribs)
Reply to "additional wikitech-l discussion re testing interactions"
Albill (talkcontribs)

I see that there is discussion on the page about having various browsers running in virtual machines.

A relatively cheap solution for this is VirtualBox, an open source project under GPLv2 for virtualization run by Oracle. This is an easy way to create VMs for Windows and various flavors of Linux (OS X is forbidden by Apple licensing requirements).

Some groups have been using VirtualBox in conjunction with Vagrant, which can deploy VirtualBox VMs through command line and/or script to set up test environments easily and then to take them down again.

I'm told that there may be licensing restrictions on using more than a set body of VirtualBox instances from Oracle but it isn't clear to me what the strictures might be and there are no docs about it on the website that I've seen.

Reply to "Virtual Machines"

Jenkins instead of CruiseControl.

Krinkle (talkcontribs)

As part of the Continuous integration project, Chad did some research to alternatives for CruiseControl.

I doesn't affect this RFC directly since Jenkins (the replacement that Chad found for CruiseControl) meets the same requirements and specification, it's just "better" and easier to use than CruiseControl.

Just thought I'd mention that hereΒ :)

Sumanah (talkcontribs)

And my understanding is that we are now using Jenkins instead of Cruise Control.

Reply to "Jenkins instead of CruiseControl."
Mglaser (talkcontribs)

I am not sure whether we should drop Selenium completely. Selenium tests do test the application as a whole. While I agree that a lot of things that are currently tested with Selenium can be substituted with QUnit or similar, I still think that selenium tests should be used to check more complex interaction patterns, e.g. to test a file uploader or GUI elements that interact via AJAX with the server.

Also, keep in mind that selenium tests are quite easy to record. While experienced developers like us probably tend to write the tests instead of recording, I think that we can actually reduce the barrier for others.

Krinkle (talkcontribs)

Both QUnit and TestSwarm are written with asynchronous and ajax support in mind. Our current test suite is in a static html-file, we're currently finishing up a to instead run these tests on Special:QUnitTest or something like that. But that's something on our end, not in QUnit.

File uploads are more complicated though, yes. But that is integration testing (or smoke testing), not unit testing. For unit testingæ' we want to and already have dropped Selenium. For integration testing, unless we find a better alternative, we should probably stick to Selenium.

Anyway, one thing at a time. Right now we're trying to clean up and get an infrastructure going on for Unit testing. Feel free to start an RFC for integration/smoke testing. There have been some alternatives, but it's an interesting topic to take next.

Reply to "Selenium's not dead :)"

request for testing documentation

Sumanah (talkcontribs)

A community member requests a document that fulfills the following:

  • Imagine being a contributor who wants to add code and test it
  • Link to internal step-by-step instructions rather than general external definitions of a term
    • (the "target group") will probably know the meaning of a term or tool - the point is how to use things in mediawiki context as a contributor
  • Add linked steps on
    • where to commit the code
    • where to commit the tests
    • what button to press on what page to start the tests (if necessary)
    • where to watch the results
    • whom to call/ask if something breaks or tests fail
  • Add infos about
    • which tools contributors can trigger themselves
    • which ones have to be started by wikimedia staff or automatically

so I'm just noting that here so people can tell me "that already exists! LINK" or improve existing documentation accordingly.

P858snake (talkcontribs)
where to commit the code
  • SVN (see below)
where to commit the tests
  • Core tests
    • Selenium tests: (SVN trunk)/selenium/ (For those that still use/play with it)
    • PHPUnit / QUnit: (SVN trunk)/phase3/tests/ (and apparently some selenium tests are back in there)
  • Extension tests
    • Should be under a folder in the extension's directory I think (although I can't find any to confirm)
what button to press on what page to start the tests (if necessary)

Don't need to (unless running local setups)

where to watch the results
  • CR, someone will comment/fixme the revisions if appropriate
  • IRC, CC spams the channel on a failed test (from phpunit tests)
whom to call/ask if something breaks or tests fail
  • Mark it in CR (if someone elses fails)
  • Attempt to fix it (preferably if its yours, or you can attempt to do others)
which tools contributors can trigger themselves
  • None (Unless they are self hosting/running the tests on their boxes)
    • CC automatically does the PHPunit tests
    • Krinkle manages the TestSwarm on the TS (JS tests)
which ones have to be started by wikimedia staff or automatically
  • Automagically
    • If CC breaks -> ^demon/Hex
Krinkle (talkcontribs)

Peachey88 wrote:

which tools contributors can trigger themselves

  • None (Unless they are self hosting/running the tests on their boxes)
    • CC automatically does the PHPunit tests
    • Krinkle manages the TestSwarm on the TS (JS tests)

I'd make that instead:

    • TestSwarm automatically does the QUnit tests (currently on TS by Krinkle, to be moved to WMF)
Reply to "request for testing documentation"
Bawolff (talkcontribs)

I think this might be slightly over-stating the problems globals cause. Well we should certainly try to decrease their presence, I don't think they're preventing newbies from contributing, or causing huge problems in developing mediawiki (although they might affect tests more than normal development).

Personally, I don't see that much of a difference between a global like $wgOut that is pretty much the only instance of its respective class and a static method to get an instance of the class to be used with the request ( RequestContext::getMain()->getOutput() or whatever)

(Of course things like $wgTitle is just evil and needs to die, but that goes without saying ;)

πŸ˜‚ (talkcontribs)

The problem with global state objects is that people can (and do) change them on a whim. By incapsulating them in accessors, you keep that sort of thing from happening and behavior is much more testable.

Krinkle (talkcontribs)

Here follows probably the worst and unlikely example.....

... consider a newbie creating a new extension and doing some temporary variable assignments before doing stuff:

$wgIn = array( 'Awesome' );

function wfMakeOut( $in ) {
  // Magic
  return $in;

$wgOut = wfMakeOut( $wgIn );
foreach( $wgOut as $out ) {
  $wgHooks['Foobar' . $out][] = 'wfMyFunc';

I know $wgOut isn't available yet in LocalSettings! like I said, bad example.

Reply to "globals"
AJim (talkcontribs)

I think you have undersold the benefits of having unit tests. Here are two more reasons to have unit tests: Unit tests document exactly how the author expects the unit to be used. Providing known working examples of use makes it easier for someone else to learn to use that unit. My personal favorite reason is that, when it comes time to revise the unit (if it stays, you will), you have a way of assuring yourself that you have not broken the externally visible behavior. I like this because it helps relieve that knot in my stomach when I have to make changes to old code.

Krinkle (talkcontribs)

I agree there are many advantages / positive side effects of having unit tests.

My personal favorite advantage to writing unit tests, is discovering bugs when throwing things at units and seeing how it responds. It's not uncommon to commit changes to units during the creation of unit tests (ie. adding return statements to a "set" function).

It also acts a bit as "intensive code review". Writing unit tests (in my opinion) is most strong when done for methods written by someone else (ie. better to have someone else write unit tests for your modules). I don't think it should be a guide line though (it's fine to write your own unit tests), but when you have the chance and are in a team, it's worth considering writing unit tests for each other. That way it's basically an intensive code review, understanding of code base and bug hunt – all three at the same time.

Reply to "advantages of unit testing"
There are no older topics
Return to "Requests for comment/Unit testing" page.