Wikimedia Release Engineering Team/Wishlist
This is the wishlist for the Release Management and QA team.
Code Deploy Dashboard
editProblem: We don't have an simple way of communicating what code is deployed to which wikis. Right now you need a combination of Gerrit and the MediaWiki Roadmap.
Stakeholders/use-cases
edit- Product Managers: I want to see one page where I can tell if code my team has written is on all Wikipedias.
- Developers: I want to know when code I have written is on the test wikis and when it'll be on English Wikipedia.
- Technical Users/Readers: I want to see if there's an obvious excuse for a problem I am seeing on my home wiki due to a recent deployment.
- Release Team: All of the above.
Solution ideas
editCreate a dashboard page that displays the currently deployed/active branches on the WMF Cluster (and, for bonus points, the Beta Cluster). The branches will link to a single page in Gerrit that lists all of the changes and their commit short message.
See this quick/"I'm not a designer" mockup from Greg.
Pieces needed
edit- Timestamp of deploys (and which group of wikis was affected, eg phase0 or phase1) (from graphite/statsd if that ever works again bug 62667)
- A way to associated a wiki group with the version that's live (see hashar's work on Special:Version parsing?)
- Super basic but informative graphs (probably something pulled from http://gdash.wikimedia.org/dashboards/reqerror/ )
- A place to put that all
Release Notes Sanity
editProblem:
13:09 < bd808> Gah! Release notes! F U release notes 13:10 < greg-g> :) 13:10 < ori-l> it's one of our "you forgot to say simon says" -1 tricks
Stakeholders:
- All MediaWiki developers - When commiting a change to MW that needs a note in the RELEASE-NOTES-X.XX file, I shouldn't have to play games with gerrit and rebasing continuously.
Solution ideas:
- rm -f RELEASE-NOTES-X.XX
- Merge conflict resolution driver
- ....
True code pipeline
editRight now code does not follow the same pipeline universally. For example:
- Code merged to master on Friday will be on the Beta Cluster for one full week before going out on the testwikis
- Code merged to master on Thursday early morning will be on the Beta Cluster for all of an hour (or a minute) before going to testwikis.
This is not ok. Code should propagate from master -> Beta Cluster -> test production -> other wikis in a consistent fashion.
Proposal
editNightly test tags - tracking: bug 65126
- Every night at a specified time we tag master in core and all extensions in use on the cluster.
- pre/post merge unit tests are run against the tag
- the tag is deployed to the Beta Cluster using multiversion - bug 65127
- browser tests are run against the beta cluster hitting the nightly tag version - bug 65128
- in the morning we review the browser tests and unit test output, determine suitability for deploy
- deploy the tag as the new wmfXX deployed version on Thursday morning
- If the latest tag is not suitable (due to failed browser tests etc) we use the last suitable nightly (eg Tues night's)
Auto-populate 'Important Changes' from RELEASE-NOTES-1.XX
editSee the 'important changes' section in most wmfXX release pages, eg this one.
It would be nice to auto-create that based on the RELEASE-NOTES file.
See the RFC: Requests_for_comment/Release_notes_automation
block commits on warnings... almost
editLet's make higher quality code. One way is to have a voting test that fails on code compile/build warnings. That's a bit extreme.
An alternative is to compare the number of warnings (and TODOs/FIXMEs?) before and after the commit and only fail if that number increases.
Performance testing environment
editSee bug 65394