Wikimedia Product/Inclusive Product Development

Background

The Inclusive Product Development Working Group is a multi-functional, multi-department group formed in an effort to align Wikimedia Foundation's (WMF) strategic vision with tactical day-to-day practice. It was formed with the high-level objective of crafting a more diverse, equitable, and inclusive product development practice.

Founding members of the working group include Ana Chang, Manager of Design Strategy; Angie Muigai, Lead Product Manager; Carol Dunn, Vice President of Product; Erika Bjune, Vice President of Engineering Technology; Jazmin Tanner, Senior Product Manager; Morten Warncke-Wang, Staff Data Scientist; and Margeigh Novotny, Vice President of Product Design & Strategy.

Current members of the working group include Jazmin Tanner, Lead Product Manager; Morten Warncke-Wang, Staff Data Scientist; Margeigh Novotny, Vice President of Product Design & Strategy; Aida Ramirez, Lead Technical Program Manager; Aishwarya Vardhana, Senior UX Designer; Cindy Cicalese, Principal Software Engineer; Kate Chapman, Director of Architecture; Naike Nembetwa Nzali, Senior Program Manager; Selene Yang, Global Diversity Equity and Inclusion Specialist, and Volker Eckl, Lead Design System Architect and accessibility specialist.

Methodology

Following best-practices, in August 2021 the Product Development Diversity and Inclusion Working Group conducted cross-functional, collaborative interviews with 15 teams across the Product and Technology departments to establish a baseline. The Working Group surfaced the processes and standards already in place; including how teams make decisions, how teams consider the users for which they build, where and how inclusive development practices are present, and power dynamics within teams. From these interviews, the Product Development Diversity and Inclusion Working Group identified areas of improvements, as well as existing solutions to share across teams. The goal was and continues to be to provide support and structure to the work the Product and Technology teams are doing, identify product development principles building on prior work, and iterate and refine the product development process.

Following these interviews, the Product Development Diversity and Inclusion Working Group summarized its learnings in the Diversity, Equity and Inclusion (DEI) Research Summary document and developed a first version (V1) framework of Product Development Best Practices. Interested teams were then identified to implement the V1 framework as part of a beta run and provide feedback in order to update and continue to refine the framework.

Objective and Key Results

The team established objectives and key results for the efforts of the working group.

Main Objective:  WMF Product* development teams build in a diverse, equitable and inclusive way.

• KR:  100% of Software and Product development teams use the DEI Product Development framework
• KR:  Collaborate across Foundation to measure DEI progress

*Product development here means all teams doing software development, not just “Product”.

Secondary Objective:  WMF embodies the spirit of open source by sharing learnings and best practices externally.

• KR: # People have read the blog post on the topics
• KR: WMF has had 2 external speaker engagements on the topic

Timeline

• July 2020: Develop interview guide for select Product and Technology teams
• August 2020: Interview teams to get a baseline understanding of existing best practices and areas of opportunity
• September 2020: Synthesize findings from Product teams and read existing best practices
• October 2020: Present findings
• November 2020 - January 2021: Share synthesized findings and bibliographies with an expert in the field to provide recommendations
• February 2021- May 2021: Develop draft Playbook and Principles
• May 2021 - June 2021: Socialize draft playbook and principles internally and externally and refine drafts based on feedback
• July 2021 - September 2021: Select volunteers to pilot the playbook
• August 2021: survey about Product Development Principles that promote Diversity, Equity, and Inclusion sent out to members of the Product and Technology departments.
• October 2021 - April 2022: Beta teams pilot playbook, providing monthly updates throughout their process
• May 2022: Share learnings via Medium article and Women in Product Conference
• June 2022: Refine playbook based on Beta team feedback
• July 2022: Incorporate feedback to develop V2 of the playbook for teams with Product Managers and share it with 2 rounds of reviewers for refinement
• August 2022: Share V2 of the playbook with all Product and Technology teams that have Product Managers, and begin interviewing Technology teams that do not have PMs to create a version of the playbook for them
• Sept 2022: V2 testing at scale kick-off with pods

How to follow along

Feel free to engage with us on our talk page.

Assigned Support Member

Below you will find which working group team member is assigned to your team.

Working Group Support Member	Teams
Volker E., Aishwarya Vardhana, Aida Alejandra Ramirez, and Natalia Rodriguez (Margeigh Novotny)	Trust and Safety, Moderator Tools, Anti-Harassment, Community Tech (Beta Team), Campaigns (Beta Team), Editing (Beta Team), Structured Data, Design Systems
Jazmin Tanner, Morten Warncke-Wang, Cindy Cicalese, and Naïké Nembetwa Nzali (Kate Chapman)	iOS, Android (Beta Team), Inuka, Web (Beta Team), Language, Growth Team (Beta Team), Search, API Platform Value, Enterprise

Artifacts

Annotated Bibliography

The working group created an annotated bibliography page to summarize readings of the group that relate to this work.

Draft Playbook

The playbook teams are stepping through to provide feedback can be found here.

Removing Bias from our Products

The presentation below was created by Jazmin Tanner to pitch a working group that would develop a standard framework for teams. The presentation was summarized into a one pager in June 2020.

Pitch deck for developing a standard framework

Product Development Principles

In August 2021, the working group carefully crafted a proposed list of Product Development Principles and reached out to the Product and Technology departments for input on how to redraft and narrow down the set of principles that would help guide our work. The survey had over 80 responses across Technology and Product.

Based on the feedback received the principles the group have landed on are:

1. Diverse teams are necessary, but not sufficient.
2. Create a culture of inclusivity.
3. Equal is not always equitable.
4. Learn from many perspectives and experiences.
5. Be intentional about who and why.
6. Build with, not for.

The explanation for these principles are:

First we ensure our teams are diverse and that the diversity is celebrated and enriched through inclusion. Then we remember there are nuances to what we are working towards i.e. equity, not equality. To do this we must learn from many perspectives and experiences while being intentional about who and why them. All this allows a team to build with, not for.

DEI Research Summary

A summary of research to date on diversity, equity, and inclusion in Wikimedia Foundation Product and Technology teams prepared by Chat Mow LLC.

A summary of research to date on diversity, equity, and inclusion in Wikimedia Foundation Product and Technology teams

Frequently Asked Questions

What is the Inclusive or DEI Centered Product Development Framework?

The framework codifies an approach to Product Development. It includes guidance and best practices for ensuring that we are both building inclusive products, as well as building in an inclusive way. It consists of guiding principles, an agile development process and the tools to help in this process.  

What work led to what I am seeing today?

This work has been discussed in multiple meetings at different times. It may be new to folks who were unable to attend meetings or new employees

• Interviewed teams August 2020
• November All Staff Meeting 2020
• Shared Report as an output of interviews January 2021
• All Hands Panel June 2021
• Wikimania Presentation August 2021
• Status update Presentation to All Staff May 2022

As a team adopting the playbook, how can we structure our feedback to be most helpful?

Every beta team should make a copy of the main Playbook and use it to check off Required and Optional items as they are completed. That’s right, the Playbook is designed to be used as a checklist.

If the team finds it’s not possible or feasible to do one of the Required items, please explain why in the Notes section. You will share your copy of the Playbook with the Working Group at the end of each phase (this is how your feedback will be collected). Also feel free to leave any other feedback in notes or comments.

The Playbook includes an open-ended notes section, to provide flexibility in feedback format for teams, but it is especially helpful if your notes and feedback include details around:

• Current work:  Were you already doing a particular practice?
• Time impact: Did adopting steps in this playbook reduce, extend, or impact timelines in any considerable way?
• Challenges: What, if any, challenges did you encounter when implementing the steps in the playbook? Please note for each specific step you had a challenge with.
• Need for resources: What resources might have helped you overcome the challenges you encountered? The working group is interested in advocating for resources where needed. This is especially important.
• Unclear definitions in the playbook: What was unclear in the playbook?  How did your team go about defining unclear definitions in the playbook? Linking to documentation, working definitions, and or external sources is especially helpful.
• Content suggestions: Was there any content you’d like to add to the playbook or anything to make it more clear and usable?
• Success: How did the product teams define success for a given project and how did this playbook serve in ensuring that success? The ideal success criteria also includes:

1. Diversifying our ecosystem
2. Promoting equity within our ecosystem
3. Making our ecosystem an inclusive place

What kind of support will be provided by the working group during team's test period of adoption?

Each Beta will be assigned a working group member to reach out to with questions and situations as they arise.

What if our team needs immediate help with the Playbook?

If your working group member isn’t available for some reason, please reach out to the #productdevelopmentframework Slack channel or comment on the talk page.

What if our team needs advice on how best to frame or facilitate difficult conversations?

Should your team need guidance about how to approach DEI topics in team discussions, or to better understand how DEI topics are being addressed at the Foundation, you can reach out to our Global DEI Team for support. You can also consult with a member of the Technical Program Management Team who can advise you on how to combine the goals you have for the conversation with the proper facilitation and meeting structure to have a successful interaction.

What is the strategic importance of growing the communities of people who have historically been left out of power and privilege globally?

See the Product Platform Strategy.

Where else can software development teams provide feedback?

In addition to the notes your team should add to your copies of the playbook, there are three other opportunities for you to provide feedback about the playbook:

• During monthly check-ins/office hours the working group will be hosting.
  • Note: the PM of the team is responsible for delivering the team's feedback at these check-ins but can send a representative in their place if they can not attend. They can leave notes in the document if they can not attend.
• Posting to the #productdevelopmentframework slack channel.
• On our talk page
• At a close-out meeting at the end of your first beta phase

As a software development team adopting the playbook, what do we do if adopting the playbook impacts initial estimates on delivery timelines?

Let us know! If/when a particular activity within the playbook takes longer than you expected, or impacts the overall timelines for your project, be sure to document this in your team's copy of the Playbook. If it’s going to impact your OKRs then let Product Management leadership know as well. In other words, the working group needs actual data such as "doing these three steps added # more hours to our normal X activity because Y.”  Until they get real feedback from teams on what this takes, the Playbook is considered a draft. We want and need real feedback.

As a team adopting the playbook, how do we ensure we fulfill all of the steps outlined as REQUIRED if resources make it difficult to complete?

The goal is trying to balance setting clear expectations around building in an inclusive way, with autonomy.  If you have feedback on whether a particular activity ought to be "Required" or "Optional", please document this opinion within your team's copy of the playbook.  

Note: the working group recognizes that some activities listed as "required" might be hard to do initially. The working group needs to know what those activities are.

As a team adopting the playbook, are there other resources or frameworks we can access to facilitate the steps outlined in best practices?

• Product Development Process
• Product Principles
• WMF Priority Emerging Markets [slides 17-22]
• Pre-Mortem Toolkit
• DACI
• How to Run a Retrospective
• Annotated Bibliography of DEI Resources
• User-Testing in target users’ preferred language (contact Design Research Team)

What does success look like?

We see this as a two fold question:

1. How does the team know they were successful in helping the Working Group
2. How does a team know they have had a positive impact in advancing diversity, equity and inclusion as a result of building under the guidance of the playbook as opposed to it being inherit ?

Success for the working group is simply receiving feedback about the process so that we can refine our framework.

To determine if the playbook had a positive influence on your ability to build for diverse audiences in an equitable, and inclusive way we will ask that your team evaluate the following:

• What parts of the playbook were you already doing?
• What parts of the playbook introduced new habits?
• What impact did your team see on your “defined key metrics”?
  • Phrased as “defined key metrics” because in going through the playbook, the KPIs are likely to change and therefore were not necessarily focusing on previously
• When reflecting on the interview responses your team provided in 2020, do they notice any changes to the answers they provided as a direct result of things in the playbook? (if applicable)

What if my team has limitations for measurement?

We ask that in the case of you having constraints of any kind that you document it in your copy of the playbook. Additionally, we encourage you to leverage the Product Analytics team’s consultation hours. Every member of the Product Analytics team (including managers) have these consultation hours regularly scheduled. They are an excellent opportunity to ask specific questions you may have, learn if there is existing data or analyses available, and discuss how to learn more about your target audiences. You may also pull data from dashboards on Superset or receive guidance from a Community Relations Specialist on how to get feedback from active community members. Finally, we encourage you to reach out to the Design Research team and schedule an appointment through the clinic calendar and contact the Wikimedia Research team to see if they have data, models, or prior research that meet your needs.

How does my team know they didn’t make things worse?

If your team has a concern that you will negatively impact a particular community, we encourage you to establish guardrail metrics. Guardrail metrics can evaluate abandonment rates, reverts and blocks. You can also monitor village pump, send out surveys, and engage with communities on project talk pages.

How does a team that isn’t creating products with a direct connection to the end user understand their impact?

A good way to understand the implications of back end development on users can be three fold:

• Gain a clear understanding from front end teams that will be using your services use cases for what they are building for and how your team can optimize services and processes to promote DEI.
  • For example, the Research team investigated if the Suggested Edits algorithm in the Android app are serving up biographies in a way that replicates the gender inequity that exists on wikis. This was broken down by language wikis. It found that biographies for men were appearing at a higher rate than women and non binary biographies, and thus would receive more improvements. A backend team creating an API to serve up biographies for a microcontribution frontend feature, has the ability and responsibility to correct biases in the underlying data by intervening in how it is delivered. In this example, this may mean explicitly setting an equal number of suggested edits for nonbinary, female, and male biographies. By not intervening in how biographies are served up to users for micro contributions the team could potentially reinforce the biases that already exist on Wikipedia. By being intentional, at the API level, where other platforms can integrate such a feature within their platform, and present the option of serving up the options in an equal or equitable manner, a backend team can promote DEI, and show the effects of such an intervention.
• Leverage things like the Web Content Accessibility Guidelines
• Leverage the playbook to revisit your practices and processes

What does DEI stand for?

For this question we will refer to the definition set by the Global DEI team at the Wikimedia Foundation:

Diversity is about the variety of identities represented in our movement, workforce, departments and teams. We intentionally focus on those identities, both inherent and socialised, that have or continue to experience direct or structural discrimination, exclusion or oppression across the globe.

• Individuals cannot be diverse; only groups can.
• We focus on identities that include but are not limited to caste, race, ethnicity, color, national origin, nationality, gender identity, gender expression, sexual orientation, age, religion, language, culture, education, abilities, income and environment.

We commit to increasing diversity, which is expressed in myriad forms.

Equity is about fair access to resources and opportunities for people by identifying and dismantling barriers that result from direct and structural systems of oppression and injustice. Equitable outcomes are realized through our behaviors, actions, policies, processes and the distribution of resources.

• Being equitable requires us to elevate those people on the margins. Equitable approaches are, therefore, contextual and can vary based on location and history. Furthermore, it requires resources to be able to do an assessment of the margins.
• It encompasses balancing and shifting years of social, political, economic, educational, and legal discrimination in an effort to promote consistent and sustained repair for communities that have and continue to experience harm.
• By eliminating barriers, we enable the full participation of groups that have experienced systemic racism/caste/colourism, sexism, and genderism; as well as other intersectional forms of oppression.
• Designing for equitable outcomes helps enhance organizational diversity and inclusive decision making.

We commit to accounting for, and responding to, shifts in power and working to redistribute it.

Inclusion is about the intentional steps that give all people, and especially those who are excluded or prevented from having a say, meaningful representation in planning and decision making across the Foundation.

• While not everyone can participate at every level of decision making, inclusion requires us to work with those impacted, or who we're designing for, to give them opportunities to share their perspectives and challenge our ideas.
• An inclusive group is necessarily diverse and leverages it by creating an environment of mutual respect, belonging, trust, support, and engagement.
• Inclusion is fundamental to ensuring the best outcomes in all physical and virtual spaces, and across all other aspects of our work.

We commit to ensuring everyone has the opportunity to share their perspectives and challenge our ideas.