Wikimedia Product Development/Product Development Process/Draft

The problem we're trying to solve

The lack of a defined process reduces understanding, increases inefficiencies and complicates dependencies and expectations between teams.

Overview

The following is a draft of the WMF Product Development Process. It describes the major Stages and Players, as well as the overall Product Release Cycle (e.g., how a feature gets from alpha to "Production"/General Availability).

There are many pieces that still need to be filled out, so please treat this as a draft.

Product Stages and Players

Product stages and players (draft)

Notes:

• “Feature” means a feature within an existing product area (product area = Editing, Mobile Web, Mobile Readers, Mobile Apps, Collaboration, etc.). There is a separate process for deciding which product areas get prioritized (to be described later, but roughly conforms to the annual and quarterly planning processes taken by WMF Product and Engineering). This process assumes that decision has been made.
• This diagram describes mainly the “as-is” state, with dotted lines representing groups that require further definition (e.g. Stakeholder, Analytics).

Introduction and concept

New ideas are generated and selected in this phase. Anyone (e.g., product owner, designers, engineers, community liaison, users) can suggest a feature idea, but the product owner decides which ones get built. While this decision is made in collaboration with the rest of the team, the product owner is the one who makes the final call.

(The following 3 phases vary in terms of how they’re implemented across teams. Some teams have relatively well defined boundaries between the phases while other teams have more porous boundaries.)

Discovery and Design

Features are taken from idea to a first pass at the user interface and functionality to be provided to the end user. Technical discovery and design also occurs to analyze different technical approaches to implementing the feature. In both UX and Technical Design and Discovery, multiple approaches are often considered, pros and cons weighed, and a decision is made regarding how to proceed.

Development

Feature gets built by engineering. Engineers (under the guidance of the tech lead) are responsible for building the feature in collaboration with UX Design and Product.

Prototype and Review

Early version of feature (usually in a staging environment as described below) is tested with end-users. Early feedback is then incorporated into subsequent designs of the feature.

Evaluation

User feedback and/or quantitative analysis is evaluated and feature is refined based on this analysis. The section below expands on this phase.

Production Launch

Once the feature has gone through the required diligence in the Beta phase (described below), progressive rollout begins. Teams use various methods for progressive rollout (e.g. in stages by batches of projects, logged in users first, releasing to an initial percentage of users and then increasing the percentage over time).

Constant monitoring of the performance metrics of this feature happens during progressive rollout. Monitoring of any unforeseen impacts on the overall system happens as well.

Evaluation

After 100% rollout, the feature is then evaluated to see if it had its intended impact. This evaluation includes both quantitative and qualitative analysis, and both short-term (e.g., days/weeks) and long-term (months/year) time frames are used. Subsequent decisions on the feature (e.g., further enhancements, rethinking the feature entirely, sunsetting the feature) are made based on this evaluation.

Maturing and support

Once in production, features require ongoing support (e.g., bug fixes, working through various issues with members of the volunteer community, etc.)

Product Release Cycle

Product Release Cycle (Draft)

Alpha Phase

The purpose of the alpha phase and alpha environments is to provide developers the ability to quickly prototype and test features for feedback. The Alpha environments are mainly for the product teams to prototype and test, and receive little exposure (if any exposure at all) to end-users.

Criteria to graduate to Beta varies from team to team. In general, the barrier to graduate from Alpha should be low (e.g., doesn't bring down other services, doesn’t destroy users’ data) to ensure that feature get in front of end-users as quickly as possible.

Examples: WMFLabs, developer’s local machine, Android Alpha build, Mobile Web Alpha (“Experimental Mode”).

Beta Phase

The various Beta programs provide a way for product teams to roll out new software in an environment where lots of end-users can use these features and give feedback to help improve the features. The aim of the teams is to get features into Beta as quickly as possible so that they may receive feedback from real users in real-world environments.

Our Beta Programs can reach tens of thousands of users (e.g., certain features within BetaFeatures on large projects) to hundreds of thousands (Mobile Web Beta)

At this phase, evaluation methods include qualitative user testing (e.g., in-person user testing, remote user testing, surveys, etc.) as well as quantitative analysis (e.g., A/B testing, funnel analysis, comparison against a baseline, etc.). These evaluation methods will be described in detail at a later point.

Examples: Android Beta build, BetaFeatures (desktop), Mobile Web Beta

Progressive Rollout

Once a feature has met the success criteria in Beta (success criteria to be expanded later), the product teams take the next step of progressive rollout. Progressive rollout begins with a first step towards 100% availability (which itself may be further defined, e.g., some features are only available to logged-in users). This could be releasing to 1% of users, releasing on a small project (or group of small projects), to logged in users only, etc. If things go well, the rollout continues in stages until 100% availability is reached. The rollout strategy should be flexible enough so that in the team can roll back easily if something goes wrong.

Examples: By project, by groups of projects, logged in only, x% of users, opt-in initially → opt-out, etc. A toolkit for progressive rollouts and will be detailed later.

General Availability

General Availability is the last step in a feature rollout. At this point, the feature is available to all users for whom the feature is intended. General Availability doesn’t necessarily mean all users will get the feature. For example, General Availability for an advanced editing feature may only be available logged-in users (or an admin feature may be available only to sysops). Rollbacks are considered only in the most acute situations.

Once General Availability is achieved, the feature is evaluated on an ongoing basis (per the previous section). Further feature enhancements may be made based on how the feature is performing.

Examples: all users, logged in users only, users of a certain permission level.