MediaWiki

Technical decision making/Technical Decision-Making Process Retrospective and Consultation plan/Retrospective and Consultation: Results and Analysis

< Technical decision making‎ | Technical Decision-Making Process Retrospective and Consultation plan

Status: Published

Authors: TDMP Retro Committee: Chris Danis, Daniel Kinzler, Kosta Harlan, Moriel Schottlender, Temilola Adeleye


This document presents the findings of the survey conducted among the Wikimedia technical community regarding their perspectives on a technical decision-making process in July 2023.

The purpose of this document is to serve as a comprehensive summary, offering insights from the retrospective collection of opinions and voices within the technical community. It is not intended to provide a new decision-making methodology or devise a fresh process.

Process edit

The survey was built and delivered through LimeSurvey with the help of the Research team.

Intended audience edit

The Technical Decision-Making Retrospective Survey requested feedback from individuals actively engaged in technical work impacting the Wikimedia environment, including those contributing code to MediaWiki and relying on production code for their tools. The audience was deliberately self-selected, engaging individuals invested in the technical decision-making process. Outreach efforts included communication through various channels such as multiple technical mailing lists, announcement in Tech News, Phabricator, and publication on MediaWiki.

Accessibility was prioritized, allowing participants to provide input through a survey, written communication publicly or privately, and/or two live Zoom listening sessions.

This multifaceted approach aimed to enable diverse perspectives within the intended audience and foster comprehensive insights into the technical decision-making landscape.

Caveat: The survey responses represent participants who are developers and individual contributors to the technical environment of the Wikimedia movement. It is important to note that other stakeholders in the technical ecosystem, such as users of the technical products and other impacted parties were not targeted in this research.

Survey Demographics edit

A total of 129 people completed the survey. Roughly the same number of surveys were begun but abandoned.

Affiliation [Q00]:

  • 67% of respondents are staff (WMF or Chapter)
  • 40% of respondents contribute as volunteers (developer or editor)
  • 10% of respondents are 3rd party professionals (developer or editor)

(Note that the numbers add to more than 100%, because some people have multiple affiliations. For instance, 13 respondents who are WMF staff also contribute as volunteer editors.)

Tenure [Q01]:

  • More than 50% the respondents have been contributing for over five years
  • Only 8% of respondent have been contributing for less than a year

Contribution [Q02]:

  • 40% of respondents contribute to WMF production code.
  • 25% indicate that they work on code that consumes public APIs.
  • 15% of respondents maintain code on-wiki or on WMF Cloud Services.
  • 12% of respondents write code for 3rd party use.
  • 7% (nine people) said that they do not contribute code. Most of these indicated that they have some kind of management role at WMF, while two were volunteer editors and one is a third-party employee.

(Note that the categories are not exclusive, as some people contribute in multiple ways)

Gender [Q03]:

  • 67% of respondents identify as “men”
  • 14% identify as “women”
  • 12% did not specify any gender
  • 11% identified as trans, non-binary, or other.

(Note that the numbers add to more than 100%, because some people identify as multiple genders)

Participation [Q15]:

  • About 40% of respondents (50 people) indicated that they have taken part in the TDMP process in the past.
  • 12% say that they were driving a decision, 25% were consulted as stakeholders, 16% were involved in the implementation.
  • 22% of respondents indicated that they have observed a TDMP process as an interested party.
  • 16 people were unsure whether they had participated or not.

Overally, very few significant differences were found between the different groups, such as staff, volunteers, third-party employees, long-term contributors, tool maintainers, and people who have participated in the process before. They are highlighted below when relevant.

Interestingly, sentiment from the 40 non-male respondents was less pronounced for all questions, with scores closer to neutral, while showing the same trends as other groups.

Analysis edit

Feedback about Current TDMP Process edit

Having a process for technical decisions is considered valuable, but there are a lot of deficiencies in the current process that need to be addressed.

Respondents consistently disagreed with the statements that the current process “meets the needs” [Q10] (effectiveness), is “open and accessible” [Q12] (accessibility), and is “easy to follow” [Q13] (transparency), ranking them at or worse than -0.4 on a scale of -2 to +2. The proposition that the “decisions are relevant” [Q11] received a neutral score close to 0.

If we examine the responses of the 50 people who have actively participated in the process [Q15], scores are significantly worse: “meets the needs” scores below -0.8, “relevant” below -0.5, “accessible” and “easy to follow” below -0.6 [Q10-Q13]. However, those who participated as drivers [Q15b] were neutral on “easy to follow”) [Q13].

Across all groups, the current process is perceived as slow (-0.6 on a scale of -2 to +2), bureaucratic (-0.6), inefficient (-0.4), non-inclusive (-0.2), and unhelpful (-0.2). Again, “decisions are relevant” is hovering at around 0 [Q16].

If we examine responses from different groups, we once again find that the scores are worse among people who have participated in the TDMP in the past [Q15]. Particularly, people who have been drivers in the process have found it to be slow and bureaucratic (both at a solid -1) [Q16].

Interestingly, of the 50 respondents who had participated in the TDMP, only 36 (72%) indicated that they have a good familiarity with the process [Q14]. Even of the 63 respondents who did not participate in the TDMP before [Q15], merely 4 (6%) said they had good familiarity.

In the free-form text responses [Q21], there were some aspects of the current process that were called out as worth keeping or in need of strengthening:

  • Keep an initial round of communication that allows stakeholders to self-identify.
  • Strengthen representation of stakeholders outside of WMF.
  • Improve clarity and transparency of the process and documentation of decisions.
  • Strengthen leadership and decision authority.

Needs for Future TDMP Process edit

There is a clear mandate for making significant changes to the TDMP process [Q17]. Only 1% of respondents want to keep the current process unchanged. The plurality of respondents (36%) even suggest to re-design the process from scratch, while another 27% say we should make minor changes (and 3% indicated they want major changes in the free-form comments). 14% indicate that they would like the process to be more like the TechCom RFC process we were using until 2020. The “other” option was used mostly to say “I don’t know”.

The results are consistent across all groups of respondents. Notably, the strongest wish for change comes from the group of people who have been driving decisions through the process [Q15]: 10 of these 16 decision drivers (63%) say that they would like to see the process re-designed from scratch, and another 5 (31%) say we should make minor changes. Interestingly, none of them want to see the process to be more like TechCom RFCs, while this idea had significant support among those who had been participating as stakeholders and observers (13% and 25%, respectively).

Across all groups, the strongest support for going back to the TechCom RFC process comes from third-party staff (33%) and maintainers of production code (20%). Support is weakest among respondents who have been around for five years or less (7%).

Overall, respondents are asking for a public process that focuses on high quality decisions that are clearly documented [Q05]. When asked to rank which features they want in a decision making process, respondents clearly went for public discussion and explicit documentation (+1.4 and +1.6, on a scale from -2 “must not happen” to +2 “must happen”). This is also supported by the free-form answers: when we asked what people most want to change about the process [Q18], 28 of 70 responses highlighted public communication and documentation.

The remaining four options got a lot less support: the idea that decisions should be made by the person or group responsible for the implementation got only weak support (+0.3), starting the process without proposing solutions did only slightly better (+0.5), closely followed by active facilitation, an owner-driven process, and open participation (around +0.6). That’s still less than half of the score received by the top two items.

There is an interesting difference between volunteers respondents and staff:

  • Volunteer contributors are generally in favor of the process being open to anyone participating at any point (+1.0) and are less supportive of decisions being made by the group in charge of implementation (+/-0).
  • Among staff contributors both ideas get lukewarm support, with open participation getting a lower score than among volunteers (+0.6) and decisions being made by implementers getting as higher score (+0.5)

The trade-off ranking of process attributes shows a similar picture [Q4]: on average, respondents from all groups favored focussing on producing the highest quality of code and design (+1.1) and including all voices (+0.7 on a scale of -2 to +2). This is confirmed by 28 of 70 responses to the free-form question [Q18] asking for a more open and inclusive process.

It is interesting to note that here, both staff and volunteer contributors support the idea of including all voices to the same degree (+0.8), while the idea that anyone can join the discussion at any point [Q05] seems more controversial, as discussed above.

Another interesting aspect is that long-term contributors valued including all voices on-par with high quality code (at +0.9), while respondents who have been contributing for five years or less value inclusivity quite a bit lower than quality (+0.1 vs. +0.4).

Looking at all groups again, respondents assigned little importance to making decisions quickly (-0.6, staff at -0.8) and to finding solutions that can be implemented with little effort (-1.2).

Respondents favor a pragmatic approach to determining at which point of decision making a formal process should be started [Q09]: 35% say the process would start when there is a need for "consultation with stakeholders”, and 32% say when there is a need to “brainstorm solutions”. Only 7% want to start immediately when a problem is identified (the status quo in the current TDMP), and only 12% want to always wait until a specific solution is proposed.

This matches the fact that respondents are generally unsure about when to engage with the process [Q08] (-0.3 on a scale of -1 to +1), while they are a little more sure of how to engage [Q07] (-0.1).

Interpretation of survey results edit

In this section, we distill insights from our survey on technical decision-making. Through a thorough analysis of participant responses, some key patterns and challenges emerge, offering valuable guidance for iterating on the decision processes.

Design tensions edit

The survey highlights some intricate trade-offs involved in crafting a decision-making process, emphasizing the need to balance simplicity, flexibility, and broad participation in future design considerations.

Adjustable process vs. Simplicity: There is a need to make the process more flexible, and offer a light-weight option. There also arose a need to make the process less complicated and elaborate. Those two may be hard to reconcile.

Wide participation vs. Efficiency: On one hand, there’s a call to make sure discussions are public and include many voices and perspectives. On the other hand, there is the weariness that decisions take a long time and involve a complex process that depends on feedback stages that slow it down.

Initiating the process edit

Many respondents preferred the option to include decision statements alongside proposed solutions during the initial phase of the process. Respondents expressed that there are cases where discussing a problem without proposed solutions is wasteful and confusing, and there needs to be room to include potential solutions if any are known.  Having a clear problem statement was seen as necessary but not sufficient to identify the group of stakeholders – since some may not be affected by the problem, but only by some of the possible solutions.

Several respondents also felt that the option of 'doing nothing' should be considered when making technical decisions. Providing insight into why a decision is necessary was seen to be as crucial as deciding what action to take. They believed that incorporating the consideration of doing nothing would ensure a thoughtful and deliberate decision-making process, aligning with the survey's emphasis on strategic and well-informed choices in technical contexts.

Explicitly considering alternative solutions early on, including the option to do nothing, was seen as likely to provide a better understanding on why the problem needs solving, and how different groups might be affected by each solution.

Accountability and authority edit

Several participants supported the idea of establishing a mechanism for addressing concerns and documenting alignment. They appear to be concerned that the decision owners are not held accountable for the consequences of their decisions. Respondents emphasized the importance of ensuring that  impacted parties have a voice in the decision.

The next shared recommendation was to Include relevant stakeholders in the decision process. The survey results emphasize the importance of empowering stakeholders from outside the team to flag a decision for consideration. Part of the purpose of the decision making process would be ensuring that people affected by a decision have a voice in making it.

Survey respondents also believed in the decision-making process until all concerns have been acknowledged, or until the decision is abandoned explicitly. Survey respondents expressed a desire for a clear and binding end to the decision-making process. There were concerns about instances where teams, growing impatient with the process, proceeded without consensus or a documented explicit decision.

Communication and process edit

Survey respondents recommended establishing a steady cadence of public updates about what decisions are in progress and what their respective status is. This was viewed as allowing people to discover decisions that are relevant to them, and would make the process more visible, accessible, and discoverable.

Participants also wanted a new process to support a clear sense of progress on technical decisions. They wanted it to be clear what is expected at any given time, what the next steps are, and who needs to take them. This came up in the free-form responses, but can also be seen from the fact that the process is perceived as slow, but on the other hand, fast decision making is not considered very important compared to other aspects.

Respondents recommended identifying stakeholders early in the process. This was seen as a way to allow internal and external groups and individuals to identify as stakeholders who can then collaborate to shape the decision. The survey indicates a wish for strengthening the role of the community in the process.

Survey participants wanted to ensure clear documentation of proposals and decisions. Many respondents recommend that the decision record should describe why a specific option was chosen, as well as which concerns and questions were addressed during the process. Similarly, respondents proposed that the initial proposal should be clear on why a decision is needed at all, and present possible solutions already under consideration.

Appendix: Survey questions edit

Demographic questions edit

Q00. What is your affiliation with Wikimedia technical spaces? (Multiple choice allowed)

  • WMF employee (57%)
  • Chapter employee (9%)
  • Volunteer developer (39%)
  • Volunteer editor (40%)
  • Third-Party developer (9%)
  • Contribute to Wikimedia as part of my job (9%)
  • Other [Free text]

Q01. How many years of experience do you have with the Wikimedia technical spaces?

  • < 1 year (8%)
  • 1-2 years (16%)
  • 2-5 years (26%)
  • 5+ (53%)

Q02. What is your interaction with technical spaces? (Choose as many as apply)

  • I contribute to code deployed by WMF (39%)
  • I maintain on-wiki code (such as gadgets) (15%)
  • I maintain bots/tools that run on toolforge or Wikimedia Cloud (16%)
  • I develop mediawiki extensions for use outside of Wikimedia (12%)
  • I write code that relies on the Wikimedia APIs (25%)
  • Other (0%)

Q03. Which of these categories describes your gender identity?

(Multiple choices allowed)

  • Male / Man (67%)
  • Female / Woman (14%)
  • Transgender (2%)
  • Non-binary (7%)
  • Genderqueer (2%)
  • I would prefer not to say (12%)
  • Other (0%)

General decision-making process edit

Q04. All decision-making processes have tradeoffs. How would you prioritize these attributes?

(The stack rank was converted to a -2 to +2 scale for evaluation, to be consistent with other questions. The option ranked lowest  by a respondent was assigned a score of -2 while the one ranked highest got a scope of +2)

  • Decisions are made quickly (-0.6)
  • Decisions are focused on the producing the highest quality of the design and code (+0.1)
  • The process involves all concerned voices (+0.7)
  • Solutions can be done by the least amount of effort and expense (-1.2)

Q05. Rank the importance of the following elements of a technical decision process  (1- must not happen; 5- must happen)

(Note that the 1 to 5 scale was shifted to a -2 to +2 scale for evaluation, to be consistent with other questions)

  • Decision owners are responsible for driving the process (+0.6)
  • There is a dedicated facilitator who ensures that the process is followed and helps to set up conversations. (+0.6)
  • Decisions start with a statement of purpose explaining a problem or opportunity, without proposing solutions (+0.5)
  • The decision is made by the person or group responsible for the implementation. (+0.3)
  • Discussions about the decision are public (+1.4)
  • Anyone can participate in the discussion at any point (+0.7)
  • The rationale for the decision is clearly documented (+1.6)

Q06. Is there anything missing from the list above that you would like to mention, either positively or negatively?

Technical Decision-Making Process (TDMP) edit

Q07. If I needed to, I know how to approach the Tech Decision Forum for a decision about something

(Conversion to scores of -1 to +1 gives an average of -0.1)

  • Yes (score+1)
  • No (score -1)
  • Unsure (score 0)

Q08. I know when (and in what types of decisions) to approach the Tech Decision Decision Making for a decision about something

(Conversion to scores of -1 to +1 gives an average of -0.3)

  • Yes (score +1)
  • No (score -1)
  • Unsure (score 0)

Q09. At what stage would you want a decision to go through the Technical Decision-Making Process?

  • Immediately when a problem is raised (7%)
  • When there is a need for brainstorming about how to approach solutions (32%)
  • When there’s a need for a consultation with stakeholders (35%)
  • When solutions are being considered (13%)
  • Other (free text) (14%)

Q10. The TDMP meets the needs of the Wikimedia technical community

(Conversion to scores of -2 to +2 gives an average of -0.44)

  • Strongly Disagree (score -2)
  • Disagree (score -1)
  • Neutral (score 0)
  • Agree (score +1)
  • Strongly agree (score +2)
  • I'm not sure (score 0)

Q11. The decisions that result from the TDMP are relevant to my contributions

(Conversion to scores of -2 to +2 gives an average of 0.0)

  • Strongly Disagree (score -2)
  • Disagree (score -1)
  • Neutral (score 0)
  • Agree (score +1)
  • Strongly agree (score +2)
  • I'm not sure (score 0)

Q12. Tech decision processes are open and accessible to affected stakeholders.

(Conversion to scores of -2 to +2 gives an average of -0.4)

  • Strongly Disagree (score -2)
  • Disagree (score -1)
  • Neutral (score 0)
  • Agree (score +1)
  • Strongly agree (score +2)
  • I'm not sure (score 0)

Q13. It is easy to stay informed about ongoing technical decisions.

(Conversion to scores of -1 to +1 gives an average of -0.5)

  • Yes (score +1)
  • No (score -1)
  • I’m not sure (score 0)

Q14. How familiar/comfortable are you with our current tech decision-making process?

(Conversion to scores of -2 to +2 gives an average of -0.1)

  • I have no knowledge of the process (score -2)
  • I’ve heard of it, but don’t know what’s involved (score -1)
  • I know a few parts of it, or the rough outline (score 0)
  • I could roughly explain it to a friend (score +1)
  • I could confidently describe it to a room full of people (score +2)

Q15. Have you ever participated in the Technical Decision-Making Process?

  • Yes (39%)
  • No (49%)
  • I’m not sure (12%)

Q15b. If your answer to the previous question was a yes, what was your role in the process?

(Percentage of those who answered “Yes” above, multiple choices allowed)

  • Decision driver (32%)
  • Stakeholder (60%)
  • Involved in the implementation of the solution (42%)
  • An interested party, not directly involved (56%)

Q16. Based on what you have observed, where do you see the Technical Decision-Making Process on the following scale?

(Evaluated as a score between +2 down to -2)

  • Streamline | Bureaucratic (-0.6)
  • Inclusive | Not participatory / open (-0.2)
  • Fast | Slow (-0.6)
  • Efficient | Inefficient (-0.4)
  • Helpful | Not Helpful (-0.2)
  • Relevant | Irrelevant (0.0)

Q17. In my opinion, the Wikimedia community should:

  • Keep the decision-making process (Technical Decision-Making Process) exactly as it is (1%)
  • Make minor improvements to the existing decision-making process based on feedback in this retro (27%)
  • Start over from scratch and design a new decision-making process (36%)
  • Change our decision-making to look more like the previous process (TechCom RFCs) (14%)
  • Something else [Free text] (23%)

Open questions edit

Q18. If I could change one thing about the existing tech decision process, it would be:

1: Encouraging an open process with involvement from volunteers and the wider community. 28
2: Less bureaucratic, more efficient, and clear decision-making process. 15
3: Clear communication and well-maintained documentation. 28
4: Clear leadership and decision authority. 12
5: Using more open platforms like Wiki and phab to facilitate the decision-making process. 8
6: Improved feedback mechanisms and tracking of decision outcomes. 11
7: No input/Unsure. 197
Total question respondents: 70

Q19. Is there a type of technical decision that lacks a process but needs one, please state it below.

1: Desire for increased involvement or participation and communication. 7
2: Technical Process and prioritization concerns. 22
3: Technical infrastructure and resources concerns. 8
4: concerns related to technical policies, procedures, and decision management. 11
5: General Concerns, Lack of Knowledge, or Experience (Including responses of "NO"). 18
6: No input/Unsure. 216
Total question respondents: 49

Q20. (If answered yes to the previous question) What would that process look like for the decision above?

1: Specific suggestions or recommendations for processes. 10
2: Desire for increased involvement or participation and communication. 5
3: Defining roles, responsibilities, or oversight in decision-making. 11
4: Reflecting satisfaction or dissatisfaction with current. 3
5: Reflecting satisfaction or dissatisfaction with previous. 1
6: No input/Unsure. 243
Total question respondents: 41

Q21. One important thing to keep from the existing tech decision process is:

1: Responses favoring certain aspects of the current process. 13
2: Expressions of dissatisfaction or disapproval with the current process. 8
3: Suggestions related to maintaining or improving stakeholder engagement. 16
4: Emphasis on documentation, transparency, and clear problem definitions. 8
5: Suggestions for process modifications for better outcomes. 5
6: No input/Unsure. 223
Total question respondents: 47

Q22. One thing that I think is missing from this survey is

1: Suggestions on how questions could be worded or framed differently. 2
2: Proposals for new questions or topics. 13
3: Feedback on the design, formatting, or structure of the survey. 3
4: Unrelated Responses / No input / Unsure. 247
Total question respondents: 27
Retrieved from "https://www.mediawiki.org/w/index.php?title=Technical_decision_making/Technical_Decision-Making_Process_Retrospective_and_Consultation_plan/Retrospective_and_Consultation:_Results_and_Analysis&oldid=6483305"