Quality Assurance/Monitoring results

Some tips to productively make sense of Jenkins browser test failure

We run Continuous integration tests on the Beta cluster and on test2.wikipedia.org. The former can be unstable since it updates to latest code every few minutes, and the latter is not running the latest code.

If there is a failure in Jenkins, you need to figure out

  • is it a transient failure on beta labs?
  • is an intermittent failure?
  • is the flaw in the test?

If the answers are "NO" then there's a bug in the master version of production code.

Once you've established the failure isn't transient or intermittent, you should manualy recreate the test scenario on beta labs or test2.wikipedia.org

(If any answer is "YES" it may be worth figuring out why there was a misleading test failure.)

Tips

edit

Open lots of browser windows!

When a test fails, the failed test run's "Build Artifacts" will have a .png that is a screenshot of the browser at the point of failure. This can be an obvious indication of the problem.

The output under "Test Result" is less helpful than the full console log (click Console log in the left-hand navigation, then click Full Log), the latter has colored output so you can scroll to the red text, and it shows all failures.

Tests may go wrong several steps before the step detecting error. It can help to watch the Saucelabs screencast