It's clear that to allow agile experimentation, not every tool or experiment would, or could, involve all these stakeholders.
Perhaps it would be useful to sketch out what experimental tools or features would look like (i.e. a *minimalist* rather than maximalist mapping of stakeholders), and what, if successful/popular, eventual "productization" of that tool/feature might look like.
I wonder if a checklist of things that *could* be done would be helpful - steps, tools and actions as a practical list. That way should a product be more lightweight, there's an understanding of experiments and steps that a team may choose to skip, but at least they would actively be choosing to do so.
The natural space for experimentation goes from Concept to Build and Release in Labs. I agree that the process should allow for experimentation without dragging the effort with overhead, and that is not clear now.
As soon as someone wants to deploy a piece of software in Wikimedia servers, things get more complicated. Maybe the checkboxes should be still in place, as an exercise of communication, even if their active involvement or approval would not be needed. "Just FYI".
If an experiment starts heading toward productization, committing resources, being prioritized in sprints and goals, aiming to be deployed in all Wikimedia projects... then those checkboxes become more important. What for a team might be "a very cool experiment" can turn into a trainwreck if the stakeholders didn't even have a chance to know that such experiment was happening.
We have precedents of cool experiments being drawn by Process, and also of experiments that turned out not to be so cool and caused crisis affecting many others.
It is becoming evident that the agility of movement between stages will be determined not by the potential stakeholders of every stage but by the checklist that will be placed at the entrance of every stage.
See for instance Phab:T120070, a task about the creation of a checklist for concept to enter the prioritization process. There it is suggested that some item of the checklist would be required, other optional. In the case of an experiment you want to run, there would be a question i.e. "Do you need funds / developers?" and the answer would be "No, thank you.", which would save to this concept a bunch a further questions related to the stakeholders that won't need to be involved.
@Ijon, are these explanations enough to resolve this topic?
Ijon's question needs some more thought. Reduced to the minimalist statement, the issue might look something like this:
It's good to do "experiments". It's good to "involve communities early". It is not always (or even usually) possible to do both of these things (well) at the same time.
If we set up "involve communities early" as the prime directive, then we are imposing significant, even destructive, burdens on experiments. We would be saying that devs may only conduct an experiment if they involve non-dev communities in it. That increases the costs (not just in money) as well as the likelihood of volunteer time being wasted on failed experiments, volunteers being disappointed by the cancellation of products that they want but which the devs need to abandon, and volunteers learning to ignore development because the cost:benefit ratio is unfavorable to them. We could adopt this, but we need to own the costs of such a position.
But if we encourage experiments, then we will frequently encounter a situation in which communities have no opportunity to be "involved early". We will have successful experiments that are presented "later", and which are rejected by some community members precisely on the grounds that these products began (as experiments) in advance of their involvement. More often, some volunteer communities may feel like discussions with them are merely for appearances, rather than true interest in their views. We could adopt this, but we need to own the costs of such a position.
IMO a better answer is a balance of the two: involving affected communities while the product is in active development, but not always (or even usually) in the experimental stage. There certainly are costs to this position, too: A "balance" means that some people will find out about projects later than they wish, some people will be asked for help much earlier than they wish, and some people will use "not early enough" as an excuse to object when the real problem is that they just don't like it or need it themselves. We could adopt this, too, but we need to own the costs of such a position.
(I also believe that it is inappropriate to have a one-size-fits-all method for involving non-dev communities, but having a process of "use your best judgment to decide when to involve communities" has all of these costs, too.)
There is IMO no perfect answer. However, I beleive we need to directly address this issue, by clearly stating a standard expectation here, regardless of whether that expectation is "You must never spend even 10 minutes thinking about an experiment before holding a formal community consultation" or "Communities can be ignored until 24 hours before the launch of any new product" or something in the sensible center of those extremes. Let's make a decision about what we're going to do, name and truly understand the disadvantages to our choice, try it out, and decide how to adjust our process from there.
@Whatamidoing (WMF) your comment is spot on! In order to draw the line, what about requiring community review only when there is a clear aim to end up deploying the project that is today an experiment. This "clear aim"could be defined by
That sounds like a practical, workable plan with concrete stages. I like it.
However, I expect some people to complain that this is "not early". We should think about ways to manage their expectations. We may want to formally and publicly define "early involvement of communities" as being equivalent to publishing information about the project on wiki at any of these three points in time.