Quality Assurance/How we prioritize

QA meeting Feb 26 -- how do we prioritize what we work on, re testing?

Attending:

  • originally: QA was Chris:
  • Michelle Grover for mobile
  • Zeljko for test automation
  • Quim for community events

Chris: overview of QA activities

Projects they've helped with: (13 entries)

  • AFTv5 - E2
  • Echo - E2
  • Page Triage - E2
  • VE
  • Mobile
  • Mobile
  • Language
  • Guided Tour - E3
  • Account Creation - E3
  • Math
  • UploadWizard
  • TMH
  • Search
  • PDBHandler - Community
  • Environments: (11 entries)

Places:

  • labs host (ee prototype)
  • beta cluster (AFT, mobile)
    • caching (Antoine says, "like 99% of our infrastructure is about caching so caching is the top root cause of issues :-]")
    • search <--- Lucene could be setup in beta
    • jobqueue
    • thumbnails
    • Lucene
    • bits.wikimedia.org
    • parsercache
    • ResourceLoader
    • swift
  • mediawiki.org (echo)
  • enwiki/prod (ve, e3)
  • Test2wiki (upload wizard, math, etc.)
  • future: vagrant

Exploratory:

  • Staff
  • Pro volunteers
  • Wiki volunteers

Automated

  • backlog!
  • needs subject matter experts (to write in plain english the testing needs)

QA Weekly goals:

E2 wants to be able to test stuff before it is merged -- beta cluster for non-master branches is being worked on.

matrix: Y - features, X- types of testing

X-axis - Types of testing:

  • test environment creation
  • Automatic
    • unit testing (qunit & Jenkins) -- skip for today
    • integration testing (Jenkins) -- skip for today
    • automated browser testing (can stop because code is bad, or test environement broken, deployment broken). More for stable stuff?
    • configuration testing (make sure it works on all browsers and operating systems)
  • Exploratory testing
    • developer/product manager exploratory testing - manual browser testing, acceptance testing
    • volunteer exploratory testing
    • QA staff exploratory testing (ex. AFT for Chris, Michelle for mobile)

Prioritization matrix: https://docs.google.com/a/wikimedia.org/spreadsheet/ccc?key=0Agte_lJNpi-OdGpJSHdMZV83UjdTdHc5UFVrdXBwaWc#gid=0 Want from features:

  • 1 exploratory testing project/month
  • 1 automated testing project/month
  • synchronize focuses with community/communications activities (blog posts, community events, etc.)

Current idea of process: build out this spreadsheet, see that it has all our engineering projects & what kinds of testing they're amenable to, and regularly check in to update it with Howie/Chris/Erik.