Wikimedia Product/Inclusive Product Development

Background
The Inclusive Product Development Working Group is a multifunctional, multi-department group formed in an effort to align Wikimedia Foundation's (WMF) strategic vision with tactical day-to-day practice. It was formed with the high-level objective of crafting a more diverse, equitable, and inclusive product development practice.

Founding members of the working group include Ana Chang, Manager of Design Strategy; Angie Muigai, Lead Product Manager; Carol Dunn, VP of Product; Erika Bjune, VP of Engineering Technology; Jazmin Tanner, Senior Product Manager; Morten Warncke-Wang, Staff Data Scientist; and Margeigh Novotny, VP of Product Design & Strategy.

Current Members of the working group include Jazmin Tanner, Lead Product Manager; Morten Warncke-Wang, Staff Data Scientist; Margeigh Novotny, VP of Product Design & Strategy; Carol Dunn, VP of Product; Aida Ramirez, Lead Technical Program Manager; Aishwarya Vardana, Senior UX Designer; Cindy Cicalese, Principal Software Engineer; Kate Chapman, Director of Architecture; Naike Nembetwa Nzali, Senior Program Manager; and Selene Yang, Global Diversity Equity and Inclusion Specialist.

Methodology
Following best-practices, in August 2021 the Product Development Diversity and Inclusion Working Group conducted cross-functional, collaborative interviews with 15 teams across the Product and Technology departments to establish a baseline. The Working Group surfaced the processes and standards already in place; including how teams make decisions, how teams consider the users for which they build, where and how inclusive development practices are present, and power dynamics within teams. From these interviews, the Product Development Diversity and Inclusion Working Group identified areas of improvements, as well as existing solutions to share across teams. The goal was and continues to be to provide support and structure to the work the Product and Technology teams are doing, identify product development principles building on prior work, and iterate and refine the product development process.

Following these interviews, the Product Development Diversity and Inclusion Working Group summarized its learnings in the Diversity, Equity and Inclusion (DEI) Research Summary document and developed a first version (V1) framework of Product Development Best Practices. Interested teams were then identified to implement the V1 framework as part of a beta run and provide feedback in order to update and continue to refine the framework.

Objective and Key Results
The team established objectives and key results for the efforts of the working group.

Main Objective:  WMF Product* development teams build in a diverse, equitable and inclusive way.


 * KR:  100% of Software and Product development teams use the DEI Product Development framework
 * KR:  Collaborate across Foundation to measure DEI progress

* Product development here means all teams doing software development, not just “Product”.

Secondary Objective:  WMF embodies the spirit of open source by sharing learnings and best practices externally.

KR: # People have read the blog post on the topics KR: WMF has had 2 external speaker engagements on the topic

Timeline

 * July 2020: Develop interview guide for select Product and Technology teams
 * August 2020: Interview teams to get a baseline understanding of existing best practices and areas of opportunity
 * September 2020: Synthesize findings from Product teams and read existing best practices
 * October 2020: Present findings
 * November 2020 - January 2021: Share synthesized findings and bibliographies with an expert in the field to provide recommendations
 * February 2021- May 2021: Develop draft Playbook and Principles
 * May 2021 - June 2021: Socialize draft playbook and principles internally and externally and refine drafts based on feedback
 * July 2021 - September 2021: Select volunteers to pilot the playbook
 * August 2021: survey about Product Development Principles that promote Diversity, Equity, and Inclusion sent out to members of the Product and Technology departments.
 * October 2021 - April 2022: Beta teams pilot playbook, providing monthly updates throughout their process
 * May 2022- June 2022: Refine playbook based on Beta team feedback
 * July 2022: Roll out modified playbook to rest of Product teams and launch Beta testing for Tech teams that do not have a PM

How to follow along
Our working group will provide bi-monthly updates on this page. Feel free to engage with us on our talk page.

Annotated Bibliography
The working group created an annotated bibliography page to summarize readings of the group that relate to this work.

Draft Playbook
The playbook teams are stepping through to provide feedback can be found here. The PDF version is also captured below.

Removing Bias from our Products
The presentation below was created by Jazmin Tanner to pitch a working group that would develop a standard framework for teams. The presentation was summarized into a one pager in June 2020.

Product Development Principles survey questions
The survey asked respondents about what their organization function was (e.g. Engineering, Product, Analytics, Design) and had an optional response about what department they belonged to. Following that, respondents were asked to rate the following set of principles on a scale of In favor/Neutral/Not in favor, with multiple choices possible (e.g. respondents could check both "In favor" and "Neutral" if they were somewhere in between):


 * 1) Be detailed, and flexible.
 * 2) Create a culture of inclusivity.
 * 3) Communication is a partnership.
 * 4) Everyone is different.
 * 5) Build with, not for.
 * 6) Be intentional about who and why.
 * 7) Team diversity supports product diversity.
 * 8) Needs and preferences change with context.
 * 9) Diverse teams are necessary, not sufficient.
 * 10) Learn from many voices.
 * 11) Equal is not always equitable.
 * 12) Decisions need reflections.
 * 13) Risk nothing, change nothing.
 * 14) Designing for a minority also benefits the majority.
 * 15) Be entrepreneurial.
 * 16) Embrace change.
 * 17) Naming exclusion unlocks the potential of inclusion.

Respondents were then given the same set of principles except "Needs and preferences change with context" with a more detailed explanation of the meaning of the principle, and again asked to rate each principle on a scale of In favor/Neutral/Not in favor. The order of most of the principles was changed from the first round.


 * 1) Be detailed, and flexible: Be clear, intentional, and detailed about who you are (and are not) building for, what their needs are and why you chose that audience. Also balance remaining flexible enough to accommodate for necessary adjustments as you learn more evidence-based information about your target users needs and the team’s capabilities.
 * 2) Decisions need reflections: Whether it is code that was pushed, a feature that was cut, a user that was excluded, or words that were said, decisions should lead to reflection. By reflecting on the decisions we make, we evaluate the positive and negative implications of said decision. That reflection should inform what, if any, steps should be taken to make adjustments based on what we’ve learned.
 * 3) Communication is a partnership: At all points of the software development process, have a two-way communication with members of our community, aspiring community members, and relevant teams at the Wikimedia Foundation. Communication should be tailored to the audience you’re engaging, expand beyond the loudest voice, start from a place of empathy, and occur often enough to inform your work. Constantly and consistently interrogate how your team can reduce barriers of entry for all to participate. Determine ways to simplify highly technical processes and reduce jargon so that anyone can learn and engage with your team at any point in your process.
 * 4) Everyone is different: We all have differences in needs and preferences. As fellow human beings we have a responsibility to make everyone feel accepted, welcome and included.
 * 5) Diverse teams are necessary, not sufficient: While having a diverse team is important for supporting product diversity, there are limits to what one individual can provide in representing a myriad of intersections. For that reason, it is important to leverage available data, tools and resources to equitably include more diverse experiences beyond the perspectives in the team. Engaging user research, Community Ambassadors, and other external resources like accessibility experts must be a built-in part of your development process to avoid unconscious gaps.
 * 6) Create a culture of inclusivity: We can’t build inclusive products if we’re not practicing inclusivity. This is not just about onboarding, we must be inclusive in every aspect of team formation, team relationships and rituals.
 * 7) Equal is not always equitable: Increasing access to products, so they are available to more users does not mean that more users will be able or willing to use those products. To make the products equitable, they need to be be modified or redesigned to accommodate the unique needs and preferences of historically underrepresented users.
 * 8) Team diversity supports product diversity: Be intentional about ensuring your team and groups you partner with represent a breadth of backgrounds, perspectives and experiences. The more diverse the team the greater the opportunity to create products that are more sensitive to the needs of more people. Leverage resources from T&C to reduce barriers during the hiring process that could deter future culture adds, and create a culture of belonging.
 * 9) Be intentional about who and why: Be clear, intentional and specific about who you are (and are not) building for... what their needs are and why you chose that audience. Balance remaining flexible enough to accommodate for necessary adjustments as evidence-based learning bring clarity about target users' needs and the team's capabilities.
 * 10) Build with, not for: To build with we must forge a win-win partnership with the target community, and practice listening deeply for their ideas, needs and preferences. We must understand the product we envision in the context of a life lived. We must constantly and consistently interrogate how our teams can reduce barriers to entry for all to participate, and we must reduce jargon to that anyone can learn and engage in the process of product development.
 * 11) Learn from many voices: It is important to leverage all available data, tools and resources to equitably include more diverse experiences beyond the perspectives in the team. To avoid unconscious gaps, engage User Research, Community Ambassadors and other external resources like accessibility experts as a built-in part of your product development process.
 * 12) Risk nothing, change nothing: Be bold, be brave, be willing to take chances.  Taking a risk often makes us uncomfortable. It's natural to go the other way. But comfort leads to complacency, complacency leads to stagnation, which is the exact opposite of change. The only other option to taking a risk is not taking any. And without taking one at all, there is no progress. That is precisely the biggest risk of all.
 * 13) Designing for a minority also benefits the majority: Products designed for underrepresented users often have additional features or functionality useful and appealing to majority users. For example, Closed Captioning was popular among people for whom English was a second language, for watching shows in which the dialog was spoken very quickly or with accents, mumbling or background noise.
 * 14) Be entrepreneurial: With the Foundation being a tech nonprofit, sometimes we have to think outside of the box and be scrappy.  Build-in time to experiment and prototype new ideas, test hypotheses, and leverage comparative analyses of industry innovations. Learn from the work of other teams and departments. We should always be learning and sharing what we’ve learned.
 * 15) Embrace change: Learn continuously, and pivot.
 * 16) Naming exclusion unlocks the potential of inclusion: Just as questions are necessary to find answers and problems are necessary to find solutions, naming exclusion is necessary to identify opportunities to make products more inclusive. When looking for ways to make products more inclusive, look at how the product is currently exclusive to underrepresented groups or individuals.

The final part of the survey was a text field where respondents could leave feedback.

DEI Research Summary
A summary of research to date on diversity, equity, and inclusion in Wikimedia Foundation Product and Technology teams prepared by Chat Mow LLC.

Milestone Updates
This is where we will provide monthly updates

What is the Inclusive or DEI Centered Product Development Framework?
The framework codifies an approach to Product Development. It includes guidance and best practices for ensuring that we are both building inclusive products, as well as building in an inclusive way. It consists of guiding principles, an agile development process and the tools to help in this process.

What work led to what I am seeing today?
This work has been discussed in multiple meetings at different times. It may be new to folks who were unable to attend meetings or new employees

+ Interviewed teams August 2020

+ November All Staff Meeting 2020

+ Shared Report as an output of interviews January 2021

+ All Hands Panel June 2021

+ Wikimania Presentation August 2021

+ Status update Presentation to All Staff May 2022

As a team adopting the playbook, how can we structure our feedback to be most helpful?
Every beta team should make a copy of the main Playbook and use it to check off Required and Optional items as they are completed. That’s right, the Playbook is designed to be used as a checklist.

If the team finds it’s not possible or feasible to do one of the Required items, please explain why in the Notes section. You will share your copy of the Playbook with the Working Group at the end of each phase (this is how your feedback will be collected). Also feel free to leave any other feedback in notes or comments.

The Playbook includes an open-ended notes section, to provide flexibility in feedback format for teams, but it is especially helpful if your notes and feedback include details around:


 * Current work:  Were you already doing a particular practice?
 * Time impact: Did adopting steps in this playbook reduce, extend, or impact timelines in any considerable way?
 * Challenges: What, if any, challenges did you encounter when implementing the steps in the playbook? Please note for each specific step you had a challenge with.
 * Need for resources: What resources might have helped you overcome the challenges you encountered? The working group is interested in advocating for resources where needed. This is especially important.
 * Unclear definitions in the playbook: What was unclear in the playbook?  How did your team go about defining unclear definitions in the playbook? Linking to documentation, working definitions, and or external sources is especially helpful.
 * Content suggestions: Was there any content you’d like to add to the playbook or anything to make it more clear and usable?
 * Success: How did the product teams define success for a given project and how did this playbook serve in ensuring that success? The ideal success criteria also includes:


 * 1) Diversifying our ecosystem
 * 2) Promoting equity within our ecosystem
 * 3) Making our ecosystem an inclusive place

What kind of support will be provided by the working group during team's test period of adoption?
Each Beta will be assigned a working group member to reach out to with questions and situations as they arise.

What if our team needs immediate help with the Playbook?
If your working group member isn’t available for some reason, please reach out to the #productdevelopmentframework Slack channel or comment on the talk page.

What if our team needs advice on how best to frame or facilitate difficult conversations?
Should your team need guidance about how to approach DEI topics in team discussions, or to better understand how DEI topics are being addressed at the Foundation, you can reach out to our Global DEI Team for support. You can also consult with a member of the Technical Program Management Team who can advise you on how to combine the goals you have for the conversation with the proper facilitation and meeting structure to have a successful interaction.

What is the strategic importance of growing the communities of people who have historically been left out of power and privilege globally?
See the Product Platform Strategy.

Where else can software development teams provide feedback?
In addition to the notes your team should add to your copies of the playbook, there are three other opportunities for you to provide feedback about the playbook:


 * During monthly check-ins/office hours the working group will be hosting.
 * Note: the PM of the team is responsible for delivering the team's feedback at these check-ins but can send a representative in their place if they can not attend. They can leave notes in the document if they can not attend.
 * Posting to the #productdevelopmentframework slack channel.
 * On our talk page
 * At a close-out meeting at the end of your first beta phase

As a software development team adopting the playbook, what do we do if adopting the playbook impacts initial estimates on delivery timelines?
Let us know! If/when a particular activity within the playbook takes longer than you expected, or impacts the overall timelines for your project, be sure to document this in your team's copy of the Playbook. If it’s going to impact your OKRs then let Product Management leadership know as well. In other words, the working group needs actual data such as "doing these three steps added # more hours to our normal X activity because Y.”  Until they get real feedback from teams on what this takes, the Playbook is considered a draft. We want and need real feedback.

As a team adopting the playbook, how do we ensure we fulfill all of the steps outlined as REQUIRED if resources make it difficult to complete?
The goal is trying to balance setting clear expectations around building in an inclusive way, with autonomy. If you have feedback on whether a particular activity ought to be "Required" or "Optional", please document this opinion within your team's copy of the playbook.

''Note: the working group recognizes that some activities listed as "required" might be hard to do initially. The working group needs to know what those activities are.''

As a team adopting the playbook, are there other resources or frameworks we can access to facilitate the steps outlined in best practices?

 * Product Development Process
 * Product Principles
 * WMF Priority Emerging Markets [slides 17-22 ]
 * Pre-Mortem Toolkit
 * DACI
 * How to Run a Retrospective
 * Annotated Bibliography of DEI Resources
 * User-Testing in target users’ preferred language (contact Design Research Team)

What does success look like?
We see this as a two fold question:


 * 1) How does the team know they were successful in helping the Working Group
 * 2) How does a team know they have had a positive impact in advancing diversity, equity and inclusion as a result of building under the guidance of the playbook as opposed to it being inherit ?

Success for the working group is simply receiving feedback about the process so that we can refine our framework.

To determine if the playbook had a positive influence on your ability to build for diverse audiences in an equitable, and inclusive way we will ask that your team evaluate the following:


 * What parts of the playbook were you already doing?
 * What parts of the playbook introduced new habits?
 * What impact did your team see on your “defined key metrics”?
 * Phrased as “defined key metrics” because in going through the playbook, the KPIs are likely to change and therefore were not necessarily focusing on previously
 * When reflecting on the interview responses your team provided in 2020, do they notice any changes to the answers they provided as a direct result of things in the playbook? (if applicable)

What if my team has limitations for measurement?
We ask that in the case of you having constraints of any kind that you document it in your copy of the playbook. Additionally, we encourage you to leverage the Product Analytics team’s consultation hours. Every member of the Product Analytics team (including managers) have these consultation hours regularly scheduled. They are an excellent opportunity to ask specific questions you may have, learn if there is existing data or analyses available, and discuss how to learn more about your target audiences. You may also pull data from dashboards on Superset or receive guidance from a Community Relations Specialist on how to get feedback from active community members. Finally, we encourage you to reach out to the Design Research team and schedule an appointment through the clinic calendar and contact the Technology Research team to see if they may have qualitative data that may meets your needs.

How does my team know they didn’t make things worse?
If your team has a concern that you will negatively impact a particular community, we encourage you to establish guardrail metrics. Guardrail metrics can evaluate abandonment rates, reverts and blocks. You can also monitor village pump, send out surveys, and engage with communities on project talk pages.

How does a team that isn’t creating products with a direct connection to the end user understand their impact?
A good way to understand the implications of back end development on users can be three fold:


 * Gain a clear understanding from front end teams that will be using your services use cases for what they are building for and how your team can optimize services and processes to promote DEI.
 * For example, the Research team investigated if the Suggested Edits algorithm in the Android app are serving up biographies in a way that replicates the gender inequity that exists on wikis. This was broken down by language wikis. It found that biographies for men were appearing at a higher rate than women and non binary biographies, and thus would receive more improvements. A backend team creating an API to serve up biographies for a microcontribution frontend feature, has the ability and responsibility to correct biases in the underlying data by intervening in how it is delivered. In this example, this may mean explicitly setting an equal number of suggested edits for nonbinary, female, and male biographies. By not intervening in how biographies are served up to users for micro contributions the team could potentially reinforce the biases that already exist on Wikipedia. By being intentional, at the API level, where other platforms can integrate such a feature within their platform, and present the option of serving up the options in an equal or equitable manner, a backend team can promote DEI, and show the effects of such an intervention.
 * Leverage things like the Web Content Accessibility Guidelines
 * Leverage the playbook to revisit your practices and processes