Data Platform Engineering/Data Products/Team Process

Team Process

These are internal Data Products team process, we describe our work intake process on a separate page.

You can track our current work on our workboard, we work in sprints - you can see our current sprint work on the right most column.

Teams Etiquettes

• If you can't make it to standup and there is no async thread yet, start an async thread in #data_products and add your updates, else add your update to async thread
• Our meetings are run in such a way that we do not expect everyone to be present, however, it should be possible for everyone to know what happened and have input / get outcomes
• Notes docs should have a standard structure defined
• Small and large decisions should be communicated to the whole team.
• All major decision must use the Decision Record template.
• Add enough description to phab tasks so anyone can pick them up. Part of show your work
• All meetings must be scheduled within one of the three overlap slots identified in our team working hours sheet
• We are circumspect about how long to spend on a problem and when to pair with others to collab on a problem
• Every patch should have "..and this is how you test and validate this change"

Task Creation Quick Links

• New Bug
• New Epic
• New User Story
• New Task

How we work

Our process is based on the Inclusive Product Development cycle. As we move from left to right we are progressively refining our understanding and narrowing scope of the work. Throughout the process there are feedback cycles allowing us to tune and pivot as we learn.

It is important that we also consider our work cycles within the context of short, medium and very long term strategy. How does what we are doing this week impact what we are doing this sprint, this year, this decade.

The small piece of work we are doing right now to close a task, user story, setup monitoring, improving CI/CD, add a test or write docs ladders up to delivering on our OKRs and overarching mission. It is crucial for us that we manage both levels of perspective, "think big, work small".

So practically speaking, how does this work?

We've looked at our approach from the Product Cycle view, the micro/macro view and now we will discuss our work process from the viewpoint of phases.

Strategise

At this point, we are defining the bounds of the problem we are trying to solve. The goal here is that we constrain the search space for solutions enough that we have a clear focus without unintentionally over specialising or biasing our work. We need to make as few presumptions as possible and focus on understanding the problem area.

In terms of time, this is a 1-3 year horizon. To place this in the context of WMF planning, the work at this phase relates directly to the Objective and/or Key Result level identification. For example, at present, Data Products is working under the Signals and Data Services(SDS) bucket and, in particular we are focused on SDS 2.5.1 and SDS 3.4.1.

The output of this stage will be analogus to an Agile initiative

Discover

The next phase is discovery and learning. In this phase we are trying to understand the problem and propose directions that we expect to take us towards our long term polestar. Part of the journey will involve iteratively refining those directions - rejecting paths we have identified as wrong and refining paths that are correct.

During Discovery, the team is spending a lot of time on spikes and research. Again to relate this to the WMF Planning process, we can view the output of this phase as similar to Hypotheses.

The output here is an Epic and the time horizon here is 1-3 months. We expect ~12 Epics for a given quarter.

Define

The transition from discovery to define has a very clear feedback loop. In this phase we are not yet in development. We are working to get our Epics to a high enough standard where we are confident both in the direction and the work involved.

We need to be able to succinctly and effectively articulate the work in order to be able to pass this phase. We will iterate on between Define and Discover until we pass our threshold.

This phase is coupled with Discover and is also in the 1-3 month timeframe. However, the output differs in that it is peer reviewed 1-pagers and Accepted Epics ready to be worked on.

Data Products is a new team, at time of writing, and we are in the process of formation. We do not yet have strict criteria for the Define phase. This will be developed over the course of FY 23/24. Last updated: 2023-10-18 by WDoran (WMF)

Plan

At this point, we have very clear Epics that are reviewed and approved for work. Now we start to break the work down into initial tasks. We break the work down using our standard templates and prepare it to be worked on in upcoming sprints. As we progress through the next phases, as described in the Product Development view, there will continue to be iteration and tuning as we learn.

Develop

Here we do the work. We plan our sprint goals, we estimate our tasks and we checkin daily. We identify blockers and issues as we go, we address them until they are no longer an issue and the work moves from left side to the right side of our sprint board.

Deliver

We release regularly, we have up to date monitoring, documentation, testing, we deliver early and often. Most importantly, we manage the expectations of those we serve and ensure they remain represented.

Maintain/Own

Depending on the output of a given cycle, it might be a tool or system that we own forever or it might be an API or pipeline that one of the groups we serve needs to own. In either case, these outputs have to fit into the budget of what our team can handle given our resources and we need to escalate that when that changes.