Wikimedia Release Engineering Team/Wishlist

This is the wishlist for the Release Management and QA team.

Code Deploy Dashboard edit

Problem: We don't have an simple way of communicating what code is deployed to which wikis. Right now you need a combination of Gerrit and the MediaWiki Roadmap.

Stakeholders/use-cases edit

  • Product Managers: I want to see one page where I can tell if code my team has written is on all Wikipedias.
  • Developers: I want to know when code I have written is on the test wikis and when it'll be on English Wikipedia.
  • Technical Users/Readers: I want to see if there's an obvious excuse for a problem I am seeing on my home wiki due to a recent deployment.
  • Release Team: All of the above.

Solution ideas edit

Create a dashboard page that displays the currently deployed/active branches on the WMF Cluster (and, for bonus points, the Beta Cluster). The branches will link to a single page in Gerrit that lists all of the changes and their commit short message.

See this quick/"I'm not a designer" mockup from Greg.

Pieces needed edit

  • Timestamp of deploys (and which group of wikis was affected, eg phase0 or phase1) (from graphite/statsd if that ever works again bug 62667)
  • A way to associated a wiki group with the version that's live (see hashar's work on Special:Version parsing?)
  • Super basic but informative graphs (probably something pulled from http://gdash.wikimedia.org/dashboards/reqerror/ )
  • A place to put that all

Release Notes Sanity edit

Problem:

13:09 <     bd808> Gah! Release notes! F U release notes
13:10 <    greg-g> :)
13:10 <     ori-l> it's one of our "you forgot to say simon says" -1 tricks

Stakeholders:

  • All MediaWiki developers - When commiting a change to MW that needs a note in the RELEASE-NOTES-X.XX file, I shouldn't have to play games with gerrit and rebasing continuously.

Solution ideas:

True code pipeline edit

Right now code does not follow the same pipeline universally. For example:

  • Code merged to master on Friday will be on the Beta Cluster for one full week before going out on the testwikis
  • Code merged to master on Thursday early morning will be on the Beta Cluster for all of an hour (or a minute) before going to testwikis.

This is not ok. Code should propagate from master -> Beta Cluster -> test production -> other wikis in a consistent fashion.

Proposal edit

Nightly test tags - tracking: bug 65126

  • Every night at a specified time we tag master in core and all extensions in use on the cluster.
  • pre/post merge unit tests are run against the tag
  • the tag is deployed to the Beta Cluster using multiversion - bug 65127
  • browser tests are run against the beta cluster hitting the nightly tag version - bug 65128
  • in the morning we review the browser tests and unit test output, determine suitability for deploy
  • deploy the tag as the new wmfXX deployed version on Thursday morning
    • If the latest tag is not suitable (due to failed browser tests etc) we use the last suitable nightly (eg Tues night's)

Auto-populate 'Important Changes' from RELEASE-NOTES-1.XX edit

See the 'important changes' section in most wmfXX release pages, eg this one.

It would be nice to auto-create that based on the RELEASE-NOTES file.

See the RFC: Requests_for_comment/Release_notes_automation

block commits on warnings... almost edit

Let's make higher quality code. One way is to have a voting test that fails on code compile/build warnings. That's a bit extreme.

An alternative is to compare the number of warnings (and TODOs/FIXMEs?) before and after the commit and only fail if that number increases.

Performance testing environment edit

See bug 65394