Wikimedia Release Engineering Team/Metrics
This page is obsolete. It is being retained for archival purposes. It may document extensions or features that are obsolete and/or no longer supported. Do not rely on any information on this page. |
This page is to start the conversation around metrics for the Release Engineering and QA team. It is most certainly a draft.
Metrics edit
Scenarios per repo edit
- probably create a short list of high priority projects/repos
- I'm not sure this makes sense. What if one repo has 100,000 LOC and another has 10,000 LOC? One repo has 20 active developers, another has 5 active developers (for some definition of "active")? One repo has a lot of WMF funding, one repo does not? Cmcmahon(WMF) (talk) 14:29, 21 July 2014 (UTC)
- 1) Every metrics needs a story (this will be my mantra) and 2) it can help identified relied-upon-but-under-resourced code. Greg (WMF) (talk) 20:04, 7 August 2014 (UTC)
- I'm not sure this makes sense. What if one repo has 100,000 LOC and another has 10,000 LOC? One repo has 20 active developers, another has 5 active developers (for some definition of "active")? One repo has a lot of WMF funding, one repo does not? Cmcmahon(WMF) (talk) 14:29, 21 July 2014 (UTC)
Browser test failure reasons edit
- Percentage due to infrastructural issues
- Percentage due to actual code breakage
- Probably requires hand coding at first
- how can we automate it? Delegate to the teams (ie: make it really easy)?
Browser test failure length edit
- # days since last green build
Pre-deploy browser test state edit
- % of red builds on the last run before the deploy (Wed night)
jenkins build times edit
- broken down by type (browser tests, unittests/linting, etc)
- broken down by 'team' (for various values of team)
- Mobile
- Flow
- VE
- MW Core
- extensions are too much work to do all of them individually, but they are too varied to get useful data out of an aggregate
- mean time to merge
- probably same list for 'teams' above
Phabricator edit
- Number of team migrated to Phabricator vs number of teams using Trello/Mingle right now
# of backport commits to deploy branches (wmfXX) edit
- Initially handcoded by Greg
- Next step: possibility (based on natural categories Greg identifies in first step): Commit Message keyword
- Future step: track the number of un-tagged backports and report that publicly
MediaWiki-Vagrant adoption edit
- How to measure this? Survey?
- We can measure labs-vagrant usage by querying wikitech (thanks, Bryan!)
Beta Cluster stability edit
- basic monitoring of uptime/response time