Phlogiston/reports by question

Monitor backlog per goal

When we look at our backlog in aggregate, we can only see the overall growth of our planned work:

If we divide our backlog by team goals, we can differentiate planned (i.e., quarterly goal) work from un-planned, which means we can measure how much more work until we reach a goal.

We can also measure the relative proportion of our work by goal and see scope creep per goal.

Benefit

Improved chances of completing our quarterly goals. Easier to say no, and to see when we aren't saying no enough.

Prerequisites

• Have defined end-points
• Divide work by category

Monitor Maintenance Fraction

We can measure the amount of our work that is not part of our quarterly goals.

Maintenance Fraction chart

Track what we've been working on recently:

Benefit

Able to balance work within the team. See how our work matches our goals.

Forecast Velocity

The Velocity forecast shows actual data as bars and forecast data as lines, based on one-week (Sunday to Sunday) snapshots. The lines represent a plausible range of values. The pessimistic forecast is the lowest three weeks of the previous three months; the optimistic forecast is the highest three; and the nominal forecast is the average of all weeks in the previous three months. (For teams using two-week Sprints or other processes in which most tasks are marked Resolved at a different cadence than weekly, this should produce weird-looking bars but accurate lines. Probably.)

In the first example, the team has high variability in weekly output, so the range of forecasts for the following week is very broad.

In the second example, the range is still very broad but a trend is emerging.

The degree to which the bars remain within the boundaries of the forecasts provide some measure of the reliability of the forecasting.

Note that Phlogiston currently calculates this every week (Sunday to Sunday). For teams that close tasks at a bi-weekly Review meeting, this misleadingly causes the pessimistic forecast to remain close to zero. Example:

Forecast completion dates and track forecasts over time.

We can forecast when we are likely to complete a given piece of work. Or, more realistically, we can identify work that is slipping indefinitely.

Velocity forecasting. Phlogiston now does simple forecasting of best, worst, and nominal velocity (best 3 weeks in last 3 months, worst 3, and average for whole 3 months) http://phlogiston.wmflabs.org/ve_tranche1_velocity_points.png

Completion forecasting. Based on velocity forecasts, this shows not only the current forecast, but a history of forecasts by week, which can give a lot of information about the reliability of the forecast (i.e, a forecast of "2 more weeks" that remains "2 more weeks" for 2 months is not a reliable forecast, whereas a forecast of "8 weeks" that becomes "4 weeks" the following months and "1 week" the month after is probably more accurate.)

Notes on how to get higher-quality forecasts:

• do progressive chunking: put in large epics immediatly and break them down over time
• re-calibrate by looking at backlog growth in past periods to better pre-set backlog size in new periods.
• smaller tasks, closing more frequently (more than 1 task per dev per week? what heuristic?)
• Do a short period of time tracking to let estimators recalibrate themselves.

Benefit

More likely to complete defined work. Limit goal-setting and other commitments based on evidence.

Identify Task data quality issues

Regularly review reports that highlight potentially incorrect or problematic data.

Data Quality Reports

Work actually completed

http://phlogiston.wmflabs.org/ve_done_count.png (TODO: replace with stable image)

Work of unknown Maintenance type

Forthcoming

Benefit

Improve the quality of tracking data.

Identify discrepencies between intentions and beliefs and reality.

Spot missed, dropped, forgotten, and otherwise unintended outcomes for tasks.

Cycle-Time Reports

These are reports that show how long work is spending in different stages of progress, such as "in testing" or "in deployment". Phabricator's built-in status field has a very limited range of status, so a full cycle-time report depends on a sequence of statuses typically built with Phabricator's projectcolumn field. These reports are not currently supported in Phabricator but have been prototyped and could be added on demand.

Benefit

Identify bottlenecks.

Measure the levels of Work in Progress to compare to optimal levels. (too much WIP = wasting time on context switching; too little WIP = running dry).