Wikimedia Product/Inclusive Product Development/Draft Playbook
In fall 2021 through spring 2022, a select number of teams (Growth, Community Tech, Android, Editing, Campaigns, Platform Engineering Team, Web) in Product and Technology elected to use version 1 (V1) of the Inclusive Product Development playbook. The playbook is intended to serve as just one of the tools teams will use to check if their practices welcome members of the community with diverse experiences, ensure their processes and practices are inclusive and equitable. The Beta teams provided feedback about V1 and that input generated a draft version 2 (V2). Draft V2 underwent two rounds of feedback by 30 people resulting in V2 of the playbook below. V2 will be used by all teams at the Foundation that have a Product Manager for the next fiscal year. We will request that teams leave notes about the steps and we will conduct a survey to gain an understanding of how their processes and impact has changed based on V2 of the Inclusive Product Development Playbook.
Technology teams will be interviewed so a version of the playbook for teams without Product Managers is created.
V2 Playbook Contents
Instructions and Guidance
|Stage ID||Stage||Instruction and Guidance||Supplementary Material(s)||Team Notes|
|0||Prework||Make a copy of this Playbook for your team to capture notes while completing the steps in the "Playbook Steps" tab. You should have a fresh playbook per project.|
|0||Prework||Before stepping through the playbook check in with the Inclusive Product Development Working Group to understand what resources are available for a team kick off and have at least one team member join the open #productdevelopmentframework Slack channel to share asynchronous insights and learn from other teams using the playbook.||Playbook Slack Channel|
|0||Prework||Encourage all team members to read the Playbook MediaWiki page and Practical Application Deck. Teams within the context of this version of the playbook refers to your core feature/development team. Usually comprised of: Product Manager, Engineering Manager, Program Manager, Designer, Engineers, Data Analyst, QA Engineer, Community Relations Specialist.||MediaWiki Page
Practical Application Deck
|0||Prework||Assign who will be responsible for each step. It is recommended that the Product Manager, Engineering Manager and Program Manager lead assigning who is responsible for tasks.|
|0||Prework||As you go through the playbook you will notice there are different stages. Below is the explanation of each stage:
Strategize: Plan for the work. Align it to the larger strategy. Do risk assessment. Articulate “why” and “for whom?”
|1||Strategize||If you are a team that isn't user facing (front-end team), throughout the playbook be sure to substitute terms like community, for front-end team. Everyone is building for someone. If you need support with scoping, reach out to your team's working group support member, which can be found on MediaWiki.||Support Team Member|
|1||Strategize||When establishing target audiences, consider selecting more than one specific partner community or group so you can see how changes may impact the groups differently.|
|1||Strategize||When defining the goals and proposals of a grant, try to make the way the goal is achieved flexible so that their is space and capacity to pivot on a solution|
|2||Discover||If the feature you build will produce edits, at the stage of reaching out to the research team with your experimentation plan, consider adding edit tags.||Edit Tags|
|2||Discover||Avoid leading questions in research. When in doubt, review research questions with Design Researcher||Leading Questions|
|3||Define||If input is part of the feature, consider multiple input modes such as voice, media and text to expand the ways in which people are able to contribute or communicate, when feasible.|
|1-3 & 6||Strategize, Define, Discover & Own||If a strategy meeting (Shallow Dive, Risk Assessment, Wrap Up meeting, Deep Dive) will include more than 30 minutes of presentation without engagement points. Consider pre-recording the presentation portion and sharing it ahead of synchronous time to allow for asynchronous thought and feedback.|
|4&5||Develop & Deliver||Before wireframes become hi-fidelity mockups, engineers should give feedback to the Designer and PM about technical limitations. Similarly, engineers should share early versions using prototyping, patch demos and other relevant tools to share with Designers and PMs early and often.|
|1-6||All||Things marked important in the importance column are steps that almost all teams should be able to do and are highly encouraged to complete. Steps marked optional are task that would benefit the team's process, but the working group recognize there may be considerable constraints in being able to accomplish it.|
|1-6||All||Throughout your development process, it is important to return to your WHY with WHO. It can be easy for your team to be pulled in many directions, remembering who you are centering in your project and why you are centering the chosen needs will help your team remain grounded and know who you may have to deprioritize. As you learn more about your WHO it is normal to change the how, what or when, but the change should always center your established target users.|
|1-6||All||For all steps in the playbook that involve more than 2 people, consider what methods are best for collecting input from a large group in an inclusive way. Work with your Program Manager to consider the best way to plan for synchronous and asynchronous touchpoints to get input across perspectives, timezones and abilities.|
|1-6||All||Where possible, allow for customization. Customization should be considered when it comes to how feedback is given/collected and for the development of a feature.||How customization can lead to inclusion|
|1-6||All||Ensure at each stage you are accounting for at least one sprint of reactive and unplanned work|
|1-6||All||When engaging with target audiences, especially from growth markets, ensure definitions are clearly understood by the target audience at a minimum and other groups that may engage secondarily|
|1-6||All||Try to limit using jargon where possible, and where not possible, ensure your team has a glossary on MediaWiki marked for translation or consult one||Team Practices Glossary|
|1-6||All||Consult the Inclusive Communication Guide on technical terminology to work towards a working environment that is safe and inclusive||Inclusive Communications Guide
Tech News Language Guide
|1-6||All||Include representations of different genders, cultures, and backgrounds whenever using images||Inclusive Communications Guide|
|1-6||All||Ensure you are following our internal Accessibility Guidance at a minimum and consider the A11y checklist project||Internal Accessibility GuidanceA11y Checklist Project|
|1-6||All||Use neutral pronouns or specific actor roles whenever possible for internal and external documentation and interfaces|
|1-6||All||Ensure you gather views and data from multiple sources. At no point should a team rely on a single team member to represent all people with whom they share an identity (tokenization).|
|1-6||All||Use a colorblind tester or simulator to diagnose any color contrast issues on mockups, features and documentation, including articles on diff, mediawiki, meta and Medium||Designing for color blindnessiOS Plugin|
|1-6||All||At each stage, surface assumptions, especially those that could lead to unintentional exclusion. I.e., assumptions about ability (age, language, technical connectivity etc).|
|1-6||All||Retrospectives should be conducted at least twice a stage or once a month, whichever is greater. If a team has regular retrospectives and topics from the project are raised during that retro, that meeting will suffice. However, if there are important topics about the project not being raised during a team's regular retrospective period, a special retro is strongly encouraged.|
|1-6||All||Allow users to sign up for direct outreach when the team is getting feedback. If users give feedback, ensure they know how the feedback will be used and stored in accordance to guidance from the privacy team. Also inform the user that while we can collect feedback in any language, responses will more than likely be in English or machine-translated when more human translation support isn't available.|
|1-6||All||Instead of using terms like "marginalized", or "global south", be specific about what audience you are targeting. Consider if you are hoping to retain "established" markets or develop with "growth" markets. Be clear about what established and growth means in your context.||Defining CommunitiesPractical Application Presentation|
|1-6||All||As you move throughout the playbook, and your project as a whole, your team should be learning, testing, questioning and experimenting. If your team learns that the direction of the project should change or a feature should be pulled, the team should feel empowered to take steps to escalate this desire and go through the steps to pivot or pull the project. Starting a project shouldn't be an excuse to finish it, especially if you discover it is the wrong course of action.|
|Stage ID||Step ID||Stage||Importance||Step||Status||Responsible||Supplementary Material(s)||Team Notes|
|1||1||Strategize||Important||Review the Organizational and Product Platform strategic plans and goals||Product Platform StrategyAnnual Plan|
|1||2||Strategize||Important||Review the Organizational and Product Platform strategic plans and goals||Product DACI|
|3||Strategize||Important||Schedule project launch (kick off) meeting or a team offsite that is intended to establish strategy and goals. Ensure all functions are involved and there are ground rules for the meeting that welcome diverse opinions early enough that dissent can be seriously considered. There should be multiple channels for feedback and communication (oral and written). Asynchronous input must be valued as significantly as synchronous input.||Consensus Decision Making
How to organize a remote offsite
Ground Rules/Working Agreements Template
Future link from Global DEI Team on planning an offsite
|1||4||Strategize||Important||Establish baseline of current users and use cases, and opportunities for equity based on your context (PLEASE READ SUPPLEMENTARY MATERIALS). Include existing research (internal and external) of the problem space you want to address.||Establishing Baseline Example 1Establishing Baseline Example 2Establishing Baseline Example 3|
|1||5||Strategize||Important||Leveraging the Wikimedia Accessibility Guidelines, set explicit accessibility goals and/or guardrails at the beginning of the project||Wikimedia Accessibility GuidelinesGuardrails Metric|
|1||6||Strategize||Important||Be intentional and clear about who you will empower and engage with (have a clear why), and what that engagement will look like (target audience). If your answer is everyone consider thinking about who you will engage with first for learning and then scale.
You should revisit what was learned in Step ID 4 when completing this step
|Tarot cards of tech|
|1||7||Strategize||Important||Establish what partners (internal and external) need to be brought into each phase and give them advance notice of the goal you intend to accomplish if not sooner. Advanced notice should occur as soon as you are aware you will need the team's support, ideally at least a quarter in advance.|
|1||8||Strategize||Important||Identify and secure dependencies on other teams/departments, ideally through program management coordination as soon as possible. If the identified team or department can't provide a service, product or support that would enable your team to accomplish your project, then the team should descope, pivot or pull the project
Note: A dependency isn't the same as a collaboration. A dependency is something that another team or department has (ex. API), that would block a project from moving forward.
|1||9||Strategize||Important||Determine potential for technical extensibility beyond your team's use case, and how the use case you are building would impact (positively and negatively) other users or teams. Work with teams and users to determine if the positive impact outweighs the negative impact or ways to limit negative impact of new or modified code.||Software Extensibility|
|1||10||Strategize||Important||Create a community engagement strategy. Teams should review the Product Guidance on MediaWiki as a starting point for how to engage the community, where applicable. Ensure your community engagement strategy includes a chorewheel or mechanism for going through user/community feedback on a regular basis. Consider requesting a community ambassador by working with your Community Relations Specialist (CRS) to reach out the manager of Community Ambassadors.
At this step, the Community Relations Specialist (CRS) is to share what the team's community engagement strategy will be for potentially selected communities (users) to relevant parties including Movement Communications. The Community Relations Specialist should also provide guidance to the team on how to ensure partner communities are not overwhelmed with the team's project plans and if adjustments to who partner communities are should be made. In cases where a team does not have a Community Relations Specialist, they should reach out to their Product Director or Group Manager for guidance on how to ensure their work and outreach efforts does not clash with other teams/departments.
|Product GuidanceMovement Communications GuideChorewheel|
|1||11||Strategize||Important||Establish “Who are we leaving out?” and be clear if you will include those that are being left out in this iteration or in the future based on your baseline understanding of "established" audiences and "growth" audiences. (What is your Strategy to Scale if and when it is appropriate)||Example of defining intentionally left out markets|
|1||12||Strategize||Important||When establishing OKRs be specific about who you are serving (based on previous steps), what assumptions you are making and how you will validate or challenge those assumptions (where possible)||Identifying Assmptions ExampleGuide for identifying assmptions|
|1||13||Strategize||Important||Get feedback from all members of the team, users, leadership and other relevant stakeholders to ensure the goals are clearly understood by all. Feedback (positive and constructive) as a reaction to the goals should be recorded and addressed before moving to the next phase. If the feedback was incorporated that should be communicated to the person(s) that gave the feedback, if it was not, the decision maker should equally share why the constructive feedback was not incorporated into the goals.||Gradients of Agreement|
|1||14||Strategize||Important||Clearly define what barriers and/or knowledge gaps you are aiming to address as well as what new opportunities you are planning to create for your target audience and why your team is uniquely positioned to do so. If your team is not uniquely positioned to address the barrier, consider what Subject Matter Experts you should consult or include to strengthen your unique positioning.||Knowledge Gaps|
|1||15||Strategize||Important||Once plans are becoming legible, conduct a Risk Assessment to identify and document risks with appropriate internal and external stakeholders||Risk Assessment|
|1||16||Strategize||Important||Iterate on plan based on risks identified in risk assessment and share evolved plan with appropriate stakeholders|
|1||17||Strategize||Important||Send an email to email@example.com to share your plans with legal if you are planning to engage with a region you have not engaged with before and ask if there are any issues/considerations for the regions you are planning to work in.||Who to go to for what in Legal|
|1||18||Strategize||Important||Create list of "Stakeholders" outside of core team and define when they should be engaged|
|1||19||Strategize||Optional||Review your plans with another PM that is not close to this work, asking them to provide feedback on if the WHO and WHY , as well as desired impact of your OKRs is clear. Also have them provide feedback on if your equity goals could be strengthened and if your engagement plan is inclusive.||DEI (Diversity Equity and Inclusion) Definitions|
|2||1||Discover||Important||Hold an early team shallow dive into the problem space. Shallow Dive should include a discussion about:
Each component of the shallow dive should discuss impacts on target audiences, take into consideration the risks surfaced from the risk assessment, baseline understandings of your target audience and impacts on equity and inclusion. Each component should also address potential impact on accessibility goals and guardrails. In some cases, engineers may need to have a follow up engineering specific meeting to think through technical limitations of the project. A shallow dive is about exploration while a deep dive focuses on coming closer to solutions. This team meeting should involve everyone involved, even contractors, interns, Ambassadors, etc. who will be significantly part of the process.
|Inclusive Design ToolkitAccessibility Guidelines|
|2||2||Discover||Important||Create project page on MediaWiki||Project Page Guidelines|
|2||3||Discover||Important||When project page is reasonably stable, reach out to Design Research to request translation support. Should Design Research lack capacity, tag project page for translation. Ensure target languages are prioritized at a minimum, while welcoming translation in all languages.||Translate Page|
|2||4||Discover||Important||Create a way to track this work that relevant parties are used to accessing. For most development teams, this usually means a Phabricator epic (e.g. 'parent' task that spawns subtasks) or project tag (or both!). There are other ways to accomplish this, though Phabricator is also accessible by the communities, making it a highly preferred option.
Ensure relevant parties are aware of the tracking mechanism. For example, this could look like subscribing them to the Phabricator epic or posting the link to a project tag on the project page.
|Phabricator GuidancePhabricator Task TemplateExample|
|2||5||Discover||Important||Collaborate with the Design Research, Research Team and Data Analytics to evolve and execute your experimentation and research plans.
If your team doesn't have an embedded CRS, share your experimentation and research plans with the CRS manager.
|Collaborating with ResearchGetting Research and Data support|
|2||6||Discover||Important||Share what has been discovered thus far through research and what you are planning to track throughout the project's lifecycle with internal and external stakeholders. Insights should be reflected on the project page, while respecting privacy guidelines and translatable||Privacy FAQs|
|2||7||Discover||Important||Start execution of community engagement strategy and establish your hypothesis in partnership with partner community members and target user groups that were defined in the Strategize phase. Surface possible biases and assumptions made in the hypothesis and community engagement strategy.|
|2||8||Discover||Optional||Consult with external organizations or groups, employee resource groups (ERGs), that focus on specific populations to increase understanding of problem space where relevant||ERGsExternal Orgs Example|
|2||9||Discover||Optional||Onboard Community Ambassadors|
|2||10||Discover||Optional||Partner with research agencies that are based in the market you are serving. Aim to support vendors founded and ran by local populations.||Supplier Diversity|
|3||1||Define||Important||Based on research, set up a deep dive meeting. The deep dive should include:
- Proposed solution and more defined scope of project with consideration for different customer journeys - How adding or taking away from proposed scope will impact time of delivery and risks (Technical and political) - High Fidelity Design Proposals with variants - Live feedback from engineers about possible technical spikes - Instrumentation considerations based on Design proposals A follow up meeting led by the engineers may need to talk place to discuss maintenance strategies and who will address maintenance should communities experience degradation. This meeting is not to be confused with the Deep Dive meetings scheduled by Product Leadership. If a Deep Dive with Product Leadership is scheduled when your team is having a Deep Dive as apart of this process, feel free to welcome them in to this meeting, but Product Leadership attendance should not be a blocker to your team meeting to discuss the topics above.
|3||2||Define||Important||Update Project Page with Designs and/or prototypes|
|3||Define||Important||Conduct Direct outreach regarding Design proposal with Community and encourage questioning of hypothesis, assumptions and proposed solutions|
|3||9||Define||Important||Review relevant existing software, gadgets and APIs|
|3||4||Define||Important||Share Designs and Instrumentation Plans with Legal, Architecture, Security and Performance if you are creating a new schema, collecting new types of data, retaining data for longer than 90 days, possibly having an impact on user performance or building a project that introduces risks for the user, community groups or organization||Instrumentation DACI|
|3||5||Define||Important||Conduct first round of usability testing with 2-3 variants of design proposals, being sure to center target audiences and measuring impact on accessibility goals/guardrails. Surface possible biases in user testing.||Checking for Bias in User TestingDesign Research Support|
|3||6||Define||Important||Share outcomes of usability testing with internal and external stakeholders|
|3||7||Define||Important||Determine which variant(s) to build or A/B test based on usability testing and technical research|
|3||8||Define||Important||Share the selected variant(s) the team intends to build or A/B test with relevant stakeholders and get feedback. Stakeholders should include (but is not limited to) other departments, target and potentially impacted audiences, design team.|
|3||10||Define||Important||Actively engage relevant stakeholders for how the feature will be built for extensibility for and/or with other people|
|3||11||Define||Important||Update measurement plan and socialize with relevant stakeholders based on DACI ensuring there is a feedback mechanism. Surface possible biases in the measurement plan. At least one person from Product Analytics should review a team's Measurement Plan and give feedback on measuring specific populations and downstream risks identified in the risk assessment before this stage is considered complete.
If a team does not have an embedded analyst, a draft of the measurement plan should be created, and the Product Manager should attend Product Analytics office hours for advice on how to update their measurement plan.
|Product Analytics Consultation HoursSample Measurement Plan|
|3||12||Define||Important||Create measurement Phab tasks||Measurement Phab Task example|
|13||Define||Important||Share measurement plan with Legal||Sample Task and Plan to share with Legal|
|3||14||Define||Important||If using Artificial intelligence, machine learning algorithms or other systems that automates decision making regarding to users or content, check training data for biases and share your project strategy and mechanisms for checking bias with the machine learning team in Technology. Your bias check should also evaluate the implications of scaling to other groups that are not presently a partner/target community (ex. different language)||How to remove bias in machine learning datahttps://github.com/EthicalML/XAI
https://dalex.drwhy.ai/#examples https://fairmlbook.org/ Language Modeling on Wiki
|3||15||Define||Important||Discuss code coverage and automated testing plan. Discuss how user stories could be covered by different types of tests.|
|3||16||Define||Important||Identify typical browser, bandwidth, platform and device usage for target audience, and develop a testing plan taking into consideration previously defined performance goals. Take extra care to consider audiences who have not been historically represented and if it makes sense to partner with other teams for appropriate coverage.|
|3||17||Define||Important||Identify if your team needs specialized skills or additional training for development.
If special skills are needed, consider one or some of the below: - working with budget holders to enroll the appropriate team member into the necessary training - schedule paired programming with someone in the Foundation that has the desired skills - request someone that has the skills be temporarily embedded on the team - hire a contractor - reconsider scope of project
|4||2||Develop||Important||Write copy and QQQ messages|
|4||3||Develop||Important||Ask ambassadors or community relations specialist to review text for translatability. Confirm that the text will be reviewed on translatewiki. Ideally features go to localization as early as possible. In the absence support from CRS, reach out to Design Research.||Localisation file format
|4||4||Develop||Important||Translate text of feature and FAQ page for feature which should be anchored to the project page||FAQ Project Page Example|
|4||5||Develop||Important||Use MediaWiki best practices and engineering architecture principles for frontend and backend development.||Architecture PrinciplesExtensions Best Practices|
|4||6||Develop||Important||Adhere to technical standards for accessibility||Technical Standards|
|4||7||Develop||Important||Design Review of live version in accordance to design standards||Design Standards|
|4||8||Develop||Important||Test concept via an Alpha, Beta or Prototype with your partner communities and those potentially impacted by feature, in their preferred language. Consult Design Research for Translation Support. Where possible, consider use of hidden deployments for testing.||How to use code snippets|
|4||9||Develop||Important||Surface technical barriers for target users (example: Performance issues in Japanese because of RestAPI). Reach out to other teams and Engineering leadership to identify solutions.||Foundational Tech Requests Board|
|4||10||Develop||Important||Incorporate feedback from Concept Testing|
|4||11||Develop||Optional||Include volunteer developers by tagging #goodfirsttask on phabricator tasks that are straightforward, not blockers, and well defined.
If your team does not have instructions on how to contribute, consider reading the Supplementary Materials to ensure your team is welcoming to volunteer developers based on the guidance from the Developer Advocacy team.
|Developer AdvocacyGood First Bug|
|4||12||Develop||Optional||Tag impacted developers on other teams and request code review|
|5||1||Deliver||Important||Design review on updated software|
|5||2||Deliver||Important||QTE Run full test of software according to test plan on Alpha or Beta. Test plan should consider bandwidth, language, color blindness, platforms, and assistive devices.|
|5||3||Deliver||Important||PM test software in a testing environment before it reaches production|
|5||4||Deliver||Important||Test instrumentation||QA Instrumentation DACI|
|5||Deliver||Important||Update Project Page and write comprehensive release notes.
Release notes for apps should go in the app update. Release notes for Web teams can look like Engineering specific updates as oppose to community updates.
|Engineering Release UpdatesCommunity Updates|
|5||6||Deliver||Important||Deploy software in production or ideally enable software when using a feature flag.|
|5||7||Deliver||Important||Test software in production according to test plan|
|5||8||Deliver||Important||Test instrumentation in production|
|5||9||Deliver||Important||Inform relevant stakeholders about deployments and ensure there are clear instructions for users to provide feedback||Quick Surveys|
|5||10||Deliver||Optional||Perform usability testing with specific populations, and people from different backgrounds|
|6||1||Own||Important||Gather and share final round of user feedback with other teams and users|
|6||2||Own||Important||Internal team sync on leading indicators and/or experiment results. Team should explicitly discuss expectations and assumptions compared to reality. This session should also discuss if any risks were triggered that were discussed in the strategy phase or if new ones were surfaced.|
|6||3||Own||Important||Share leading indicator results and/or experiment results with all stakeholders and community while respecting privacy guidelines. It is also recommended that results are shared during the Monthly Product All meeting. Please reach out to firstname.lastname@example.org to get your "Product Updates" on the agenda.|
|6||4||Own||Important||Follow up to ensure data collection ends or maintenance in accordance to policy and team needs|
|6||5||Own||Important||Establish a “future work” project page or Phab Epic where anyone can add insights/learnings and feature proposals for the future that may not be worked on more immediately. When establishing this space, clearly define what future means for your team and the process in which you will triage incoming requests.|
|6||Own||Important||Conduct a project wrap-up meeting with stakeholders. The wrap-up meeting should evaluate who was left out and why, the impact of exclusion (intentional and unintentionally), what users the feature resonated with the most and why, and follow up plans for scaling or removing feature in the future.|
|6||7||Own||Important||Follow up on plans for scale or feature removal based on the project wrap-up. Ensure plans account for localization.|
|6||8||Own||Important||Triage post-deployment remaining tickets and clean up feature flags or incomplete code|
|6||9||Own||Important||Create plan for future technical/design debt and maintenance. At this step, technical documentation should be comprehensive enough that another team can pick up where the original team left off.|
|6||10||Own||Important||Document and elevate awareness of how goals shifted over time via the project page, and why. Also transparently share plans for the future and engage target audiences.|
|6||11||Own||Optional||Launch a sprint for technical and design debt|
|6||12||Own||Optional||Share your learnings with other teams/departments and propose changes/additions to Inclusive Product Development Playbook|
|6||13||Own||Optional||Surface to leadership what you weren't able to accomplish due to limitations of any kind and what work arounds you had to leverage instead||Quarterly Learning Sessions|
V1 Beta Teams Test Results
In May 2022, our Beta testing teams completed a survey and turned in their copy of the playbooks. The information from the surveys and playbooks were synthesized and an external consultant shared recommendations based on this feedback.
Below are the themes shared between teams.
Time vs Deadlines
Over time it is common for teams to work toward optimization, streamlining their process in order to hit deadlines. Common as it may be, this individualized approach often lacks resilience and as a result of incremental adjustments, teams can become increasingly divergent in the ways that they get their work done. While this isn’t always a bad thing, it can certainly present challenges when attempting broad adoption of new tools and methodology.
Teams that introduced the Playbook earlier in the process seemed to have an easier time adding it into their workflow. Teams that were further along in the process had to pause, and rework their goals in order to include the playbook’s recommended steps.
The playbook helped guide team conversations, but left teams feeling unsure about how much time to invest in specific tasks, versus moving on to the next step required to meet their larger project goal. Teams questioned the rigidity of the process, wondering if all tasks must be completed before they could move to the next phase.
There was a shared belief that the approach defined in the playbook would require much more time. For instance, the time required to define and recruit a representative group of participants takes time. Translating between all relevant languages takes time (and costs money). The more you focus on people and give them time to engage, it all takes more time. This slower pacing can feel at odds with more agile ways of working, leaving people to ask what does a Prototype/MVP look like with this new approach? Not all of the 111 tasks in the playbook require the same level of investment - some tasks seem deceptively simple, only to end up occupying far more time than expected. Providing more guidance around WMF’s expectations for time spent within each stage in the process would help teams more accurately scope timelines.
Clarity & Specificity
In many cases, concerns around time came back to a lack of clarity around the expectation or specificity of the task itself. From determining which guidelines to follow or knowing who to ask if you got stuck, to knowing if you’ve completed a task to expectations, all of these issues could be clearer, if not specified on a task-by-task basis.
Many of the tasks have been distilled down to their simplest form, which puts a lot of work onto the plate of team members looking to unpack the meaning of each bullet point, without always feeling certain that they’ve unpacked it correctly.
Requesting that a team consult with a group, or adhere to a best practice without some further guidance and definition leaves a lot of room for interpretation, and continues the challenge of teams developing their own divergent approach to getting the work done.
Many teams expressed questions related to knowing how their work would be measured. Teams wanted to know how they would be reviewed, in order to learn and improve for next time. Some teams went so far as to request that the tasks be more binary or quantifiable, presumably so that they could self-assess their own performance with the playbook.
While the playbook did provide some direction for folks trying to find answers to their questions, the process of bringing someone in and getting them up to speed can feel like a significant time sink.
The majority of the following recommendations are designed to address the challenges outlined in the themes above. In many cases, when addressing an issue of specificity there is a direct benefit to the concerns around time requirements.
Ensure that every task in the Playbook is as specific as possible.
The more detail you can provide in the task the better, but if that isn’t possible either link to the details or a resource that makes it easier for someone to complete the task in a timely manner.
Clearly define the desired outcome for each task.
If the goal is to have every WMF project be WCAG 2.1 AAA compliant, be clear that will be how success is determined. Checklists are best suited to Yes/No questions, but if a task isn’t as simple as yes or no, help people know they’re doing the right thing by clarifying the end goal of the task.
Conduct a time audit to ensure that all tasks are achievable within a reasonable period of time.
Reviewing each of the 111 tasks in the playbook and assigning each with an estimate for the time required complete would be helpful when thinking about each task’s impact to project timelines. A time audit will be much easier after all of the previous recommendations have been addressed. Reducing ambiguity will remove significant levels of uncertainty, which in turn reduces hesitation, debate, and the need to search for answers.
Sort the binary (quantifiable) tasks and the mindset or qualitative tasks.
This isn’t always clear, even the WCAG 2.1 checklist starts with internalizing a set of four principles. Having open-ended items (like principles) alongside more quantifiable tasks introduces ambiguity and uncertainty. How do you know when to move on to the next item - if you’re not sure you’ve sufficiently or accurately internalized an ‘inclusive mindset’? A particular mindset or deeply internalized principle can be a powerful thing, and essential for guiding someone through the ambiguity of a project, but it takes time to shift your way of thinking and working, and that time rarely fits within the scope of a single project.
Clarify WMF time expectations to ensure a more equitable product design process.
If a team doesn’t know the foundation’s expectations around time, they will resort to the product priorities as they have in the past. If WMF states that a minimum of 20% of every project timeline should be dedicated to explicitly ensuring more equitable product outcomes, that prioritizes the playbook over business as usual.
Reconsider the rollout plan for the playbook.
Rather than introducing the playbook as part of the project kick-off, align the phases of the playbook to the specific project timeline or milestones. Some teams seem to have done this already, the difference would be having someone from the DEI Playbook team with them at each stage. This phased rollout would provide an opportunity for a DEI Playbook team member to be present for reflections on the previous steps, as well as contextualizing the playbook and answering questions to guide the project team into the next stage.
Provide more contextual support to the project teams.
The Wikimedia Product/Inclusive Product Development page is a useful resource but even so, the teams had questions about how best to tackle a task, or who to ask when a team got stuck on something. If the playbook included a list of support resources for each phase, including experts' names and contact, links to relevant documents, link to slack support channel, and the ability to @ the Playbook team in their document. The project team PM should also have a series of brief regularly scheduled standups with a member of the DEI playbook team, this time can be used to help answer team questions or provide feedback and suggestions for helping the team be successful in meeting the goals defined by the playbook.
Within the recommendations above, there are hints and references to materials that don’t exist or may not exist in a way that fully addresses the need. This section includes some of the materials that will provide more support to teams as they work through the playbook. These materials range from role assignments to digital collections.
- Playbook Marketing Campaign
- Playbook v2
- Slack Support as a Community of Practice
- Examples, Case Studies & Templates
- Resource Library
- Open Office Hours
V1 Playbook Contents
Plan for the work. Align it to the larger strategy. Do risk assessment. Articulate “why” and “for whom?”
- Involve all functions (Product Management, engineering, design, community, etc) in setting strategy, including analytics
- Welcome diverse opinions on what the plan should be, by allowing time for dissent, listening to dissent and allowing multiple channels of communication for feedback (both oral and written)
- Specify DACI for decision-making for different types of decisions
- Ask what partners need to be brought into each phase
- Understand the Organizational and Product Platform strategic plans and goals
- Explicitly discuss DEI as part of the plan
- Consider a wide variety of use cases
- Set explicit accessibility goals at the beginning of the project
- Be intentional about choosing which communities to engage with (have a clear why)
- Coordinate the choice of communities with other teams
- Ask, “Who are we leaving out?”and incorporate either outreach or checkpoints later in the process
- Be intentional and explicit in your OKRs about who, including DEI centered OKRs
- Get feedback on clarity of goals and strategic alignment of these goals with the team, users, expert insight, and stakeholders
- Share plans with other teams and leadership
- Conduct a Pre-Mortem to identify risks
- Iterate on plan based on risks identified in Pre-mortem
- Strategy and planning are constantly evolving through learnings- recognize that the plan may change. Discuss this with the team.
- Check in with legal on any issues/considerations for the regions you are planning to work in.
- Review the Community Engagement Guidelines as a team
- Review your plans with another PM, asking them to review it for DEI objectives.
- Incorporate user research when creating your plans
- Plan Discovery research with risk factors from Pre-Mortem in mind
- Use data when creating your plans
- Have a team offsite when starting a new project--focused time is valuable.
- Plan for reactive work in order to ensure space for proactive work.
- Plan to use the Best Practices Playbook for every stage of the process.
- Create the space for differing opinions--ask others “what should be different and why?”
- Review your heuristics for choosing partner wikis
- Ask how you will champion specific populations as you move through the development process
- Share the context of your DEI-related goals and plans with others
- Have a team mission that explicitly states your intentions around developing in an intentional DEI-aware way.
- Talk with people in historically underrepresented communities to find out what they think about existing products, services, or brands similar to yours.
The goal of this phase is to identify the problems you’ll be solving for. This includes identifying users you’ll be working with, understanding the needs and goals they have and how these experiences fit into their lives.
- Identify the problems you are trying to solve--user problems and needs, product problems and inclusion problems
- Consult with user research on the problems you are solving for
- Provide context for any partner teams, including tech teams, that you are working with; help them understand the user experience and potential impact of the work.
- Ask “who else?” as you define the work; be specific. Eg, How might this problem look different across different socio-economic groups? Who might not be able to use this product and why?
- What is the WCAG 2.1 level AA guidance in this area?
- At the end of this phase, submit your Playbook for the project to [email alias here] 
- Avoid leading questions in research, review research questions with Design Researcher before conducting research
- Consult with external organizations/DEI experts that focus on specific populations to increase learning
- Share insights through written documentation so we have institutional memory of our learnings
- Share with other functions/partners/departments (marketing etc)
- Partner with research agencies that are in the market you are serving
 The checklist in this Playbook will be our internal mechanism for tracking the kinds of practices, behaviors and activities our teams are using. We will be comparing what we learn via this reporting channel to the baseline research we did at the beginning of this process in order to track progress toward a more DEI-centered product development process.
The goal of this phase is to turn research, insights, etc into an addressable and achievable scope of work with a concise problem statement.
- Evaluate all of your decision heuristics for biases
- Develop your hypotheses with your partner communities
- Gather feedback on definitions from partner communities, including underrepresented populations
- Use neutral pronouns or specific actor roles whenever possible in the documentation
- Are any images used representative of different genders, cultures and backgrounds?
- Are multiple input modes such as voice and text available if data input is part of the feature?
- Documentation should be easily understandable to laypeople, free of jargon
- If using AI or ML algorithms, then check training data for biases
- Work with analytics on how to measure impact, including specific populations
- Ensure you gather views and data from multiple sources instead of relying on a single team member to represent all people with whom they share an identity. Avoid tokenization, including for user personas.
- Define code coverage and any needed unit testing plan to be executed as part of the development cycle
- Identified typical browser and device usage for targeted users, including underrepresented communities, and develop a testing plan
- Identify if your team needs specialized skills or additional training for development
- Examine assumptions that could lead to exclusion. I.e., assumptions about ability (age, language, technical connectivity etc).
- Use common language around DEI efforts
- Question your hypothesis: how will you measure underserved populations?
- Share definitions publicly to get feedback
- Identify and question the assumptions about the feature solutions.
- Identify different Customer Journeys that reflect multiple viewpoints
- Use a colorblind tester or simulator to diagnose any color contrast issues.
- Evaluate and refine your hypothesis through Causal Loop Diagrams
- Hold a design sprint focused on bringing in different viewpoints
- Review data models for inclusivity and privacy
- Consider how legacy terminology in policies (and elsewhere) might require less “insider” synonyms and simplify language where possible 
 For example “revert” is a technical way of expressing “undo” – which might be a more layperson-friendly way of expressing the same concept. Consider when a legacy term might need to be challenged with the community for the purpose of making the platform more accessible to newcomers.
Development of experiences identified in the STRATEGIZE phase, with refinement and further development of the most successful concepts in preparation for the DELIVER phase. This includes design iterations.
- Look for bias in underlying tools, systems such as AI or ML. Work with the responsible teams to address the bias
- Develop multiple concepts
- Adhere to accessibility design and technical standards
- Test concepts with your partner communities, in their preferred language
- Incorporate community feedback
- Iterate on the most promising designs
- Design according to design standards
- Use MediaWiki best practices and engineering architecture principles for frontend and backend development. (Vue etc)
- Design with accessibility goals in mind
- Conduct concept testing in the native language of the partner communities you are working with
- Conduct concept testing with your target populations
- Community liaison PR effort to socialize the work to target populations
- What technical barriers are there to inclusion? Reach out to other teams and Engineering leadership to identify solutions.
- Ask for volunteer developers support
- Conduct concept testing across a variety of populations and languages
Testing and delivery of final code and/or interface. This could be either for delivering a smaller scaled test or final scaled experience to production.
- Perform user testing with specific populations, and people from different backgrounds
- Beta test with people from different backgrounds, genders, ages and cultures
- Conduct user testing in the native language of the partner communities you are working with
- Evaluate performance for low bandwidth users
- Test assistive devices
- Do we have standard devices that we develop and test on that should be included here?
- Instrument for downstream risk factors identified in the planning phase
Ongoing monitoring of positive and negative impacts resulting from the product/code delivered.
- Do retrospectives on your work
- Who was left out of previous work and why? Is there input into future strategy?
- Plan for future technical/design debt around DEI goals
- Gather user feedback and share it with other teams and users
- Make the results of your work public
- Consider language barriers in feedback--ask community ambassadors, Community Relations Specialists, and research for help in solving this.
- Work with analytics on how to measure specific populations over time as needed
- Work with analytics on ensuring privacy through metrics. Question if you need to keep measuring specific metrics.
- Elevate awareness of how goals shifted over time, and why
- Evaluate real results compared to expected
- Have a plan for localization as you scale
- What is community feedback?
- Track risk factors identified in STRATEGY phase
- Responsible and respectful exit from intensive community engagement
- Create a “chore wheel” to gather and share user feedback
- Throughout this process, have a “future work” document where you can collectively write down insights/learnings for the future.
- Establish user or community councils for feedback
- Use user feedback surveys
- Share your DEI Prod Developments efforts publicly to get feedback
- Identify unexpected uses
- Identify unexpected resultsIdentify the audience actually using (and not using) it. Identify why.