Wikimedia Product/Inclusive Product Development/Draft Playbook

From mediawiki.org

BACKGROUND[edit]

As of fall 2021, a select number of teams in Product and Technology are using the playbook below to step through each stage of the Inclusive Product Development Framework. This playbook is just one of the tools teams will use to check if their practices welcome members of the community with diverse experiences, ensure their processes and practices are inclusive and equitable.

Steps in the inclusive product development framework in a loop (strategize, discover, define, refinement, feedback, and own
Steps in the inclusive product development framework

STRATEGIZE  [edit]

Plan for the work.  Align it to the larger strategy.  Do risk assessment.  Articulate “why” and “for whom?”

REQUIRED[edit]


  • Involve all functions (Product Management, engineering, design, community, etc) in setting strategy, including analytics
  • Welcome diverse opinions on what the plan should be, by allowing time for dissent, listening to dissent and allowing multiple channels of communication for feedback (both oral and written)
  • Specify DACI for decision-making for different types of decisions
  • Ask what partners need to be brought into each phase
  • Understand the Organizational and Product Platform strategic plans and goals
  • Explicitly discuss DEI as part of the plan
  • Consider a wide variety of use cases
  • Set explicit accessibility goals at the beginning of the project
  • Be intentional about choosing which communities to engage with (have a clear why)
  • Coordinate the choice of communities with other teams
  • Ask, “Who are we leaving out?”and incorporate either outreach or checkpoints later in the process
  • Be intentional and explicit in your OKRs about who, including DEI centered OKRs
  • Get feedback on clarity of goals and strategic alignment of these goals with the team, users, expert insight, and stakeholders
  • Share plans with other teams and leadership
  • Conduct a Pre-Mortem to identify risks
  • Iterate on plan based on risks identified in Pre-mortem
  • Strategy and planning are constantly evolving through learnings- recognize that the plan may change.  Discuss this with the team.
  • Check in with legal on any issues/considerations for the regions you are planning to work in.

SUGGESTED[edit]


  • Review the Community Engagement Guidelines as a team
  • Review your plans with another PM, asking them to review it for DEI objectives.
  • Incorporate user research when creating your plans
  • Plan Discovery research with risk factors from Pre-Mortem in mind
  • Use data when creating your plans
  • Have a team offsite when starting a new project--focused time is valuable.
  • Plan for reactive work in order to ensure space for proactive work.
  • Plan to use the Best Practices Playbook for every stage of the process.
  • Create the space for differing opinions--ask others “what should be different and why?”
  • Review your heuristics for choosing partner wikis
  • Ask how you will champion specific populations as you move through the development process
  • Share the context of your DEI-related goals and plans with others
  • Have a team mission that explicitly states your intentions around developing in an intentional DEI-aware way.
  • Talk with people in historically underrepresented communities to find out what they think about existing products, services, or brands similar to yours.

DISCOVER  [edit]

The goal of this phase is to identify the problems you’ll be solving for.  This includes identifying users you’ll be working with, understanding the needs and goals they have and how these experiences fit into their lives.

REQUIRED[edit]


  • Identify the problems you are trying to solve--user problems and needs, product problems and inclusion problems
  • Consult with user research on the problems you are solving for
  • Provide context for any partner teams, including tech teams, that you are working with; help them understand the user experience and potential impact of the work.
  • Ask “who else?” as you define the work; be specific.  Eg, How might this problem look different across different socio-economic groups? Who might not be able to use this product and why?
  • What is the WCAG 2.1 level AA guidance in this area?
  • At the end of this phase, submit your Playbook for the project to [email alias here] [1]
  • Avoid leading questions in research, review research questions with Design Researcher before conducting research

SUGGESTED[edit]


  • Consult with external organizations/DEI experts that focus on specific populations to increase learning
  • Share insights through written documentation so we have institutional memory of our learnings
  • Share with other functions/partners/departments (marketing etc)
  • Partner with research agencies that are in the market you are serving

TEAM NOTES[edit]


[1]    The checklist in this Playbook will be our internal mechanism for tracking the kinds of practices, behaviors and activities our teams are using. We will be comparing what we learn via this reporting channel to the baseline research we did at the beginning of this process in order to track progress toward a more DEI-centered product development process.

DEFINE[edit]

The goal of this phase is to turn research, insights, etc into an addressable and achievable scope of work with a concise problem statement.

REQUIRED[edit]


  • Evaluate all of your decision heuristics for biases
  • Develop your hypotheses with your partner communities
  • Gather feedback on definitions from partner communities, including underrepresented populations
  • Use neutral pronouns or specific actor roles whenever possible in the documentation
  • Are any images used representative of different genders, cultures and backgrounds?
  • Are multiple input modes such as voice and text available if data input is part of the feature?
  • Documentation should be easily understandable to laypeople, free of jargon
  • If using AI or ML algorithms, then check training data for biases
  • Work with analytics on how to measure impact, including specific populations
  • Ensure you gather views and data from multiple sources instead of relying on a single team member to represent all people with whom they share an identity.  Avoid tokenization, including for user personas.
  • Define code coverage and any needed unit testing plan to be executed as part of the development cycle
  • Identified typical browser and device usage for targeted users, including underrepresented communities, and develop a testing plan
  • Identify if your team needs specialized skills or additional training for development

SUGGESTED[edit]


  • Examine assumptions that could lead to exclusion.  I.e., assumptions about ability (age, language, technical connectivity etc).  
  • Use common language around DEI efforts
  • Question your hypothesis: how will you measure underserved populations?  
  • Share definitions publicly to get feedback
  • Identify and question the assumptions about the feature solutions.
  • Identify different Customer Journeys that reflect multiple viewpoints
  • Use a colorblind tester or simulator to diagnose any color contrast issues.
  • Evaluate and refine your hypothesis through Causal Loop Diagrams
  • Hold a design sprint focused on bringing in different viewpoints
  • Review data models for inclusivity and privacy
  • Consider how legacy terminology in policies (and elsewhere) might require less “insider” synonyms and simplify language where possible [1]

TEAM NOTES[edit]


[1]    For example “revert” is a technical way of expressing “undo” – which might be a more layperson-friendly way of expressing the same concept. Consider when a legacy term might need to be challenged with the community for the purpose of making the platform more accessible to newcomers.

DEVELOP  [edit]

Development of experiences identified in the STRATEGIZE phase, with refinement and further development of the most successful concepts in preparation for the DELIVER phase.  This includes design iterations.

REQUIRED[edit]


  • Look for bias in underlying tools, systems such as AI or ML. Work with the responsible teams to address the bias
  • Develop multiple concepts
  • Adhere to accessibility design and technical standards
  • Test concepts with your partner communities, in their preferred language
  • Incorporate community feedback
  • Iterate on the most promising designs
  • Design according to design standards
  • Use MediaWiki best practices and engineering architecture principles for frontend and backend development.  (Vue etc)
  • Design with accessibility goals in mind
  • Conduct concept testing in the native language of the partner communities you are working with
  • Conduct concept testing with your target populations
  • Community liaison PR effort to socialize the work to target populations

SUGGESTED[edit]


  • What technical barriers are there to inclusion? Reach out to other teams and Engineering leadership to identify solutions.
  • Ask for volunteer developers support
  • Conduct concept testing across a variety of populations and languages

TEAM NOTES[edit]


DELIVER  [edit]

Testing and delivery of final code and/or interface.  This could be either for delivering a smaller scaled test or final scaled experience to production.

REQUIRED[edit]


  • Perform user testing with specific populations, and people from different backgrounds
  • Beta test with people from different backgrounds, genders, ages and cultures
  • Conduct user testing in the native language of the partner communities you are working with
  • Evaluate performance for low bandwidth users
  • Test assistive devices
  • Do we have standard devices that we develop and test on that should be included here?

SUGGESTED[edit]


  • Instrument for downstream risk factors identified in the planning phase

TEAM NOTES[edit]


OWN  [edit]

Ongoing monitoring of positive and negative impacts resulting from the product/code delivered.

REQUIRED[edit]


  • Do retrospectives on your work
  • Who was left out of previous work and why?  Is there input into future strategy?
  • Plan for future technical/design debt around DEI goals
  • Gather user feedback and share it with other teams and users
  • Make the results of your work public
  • Consider language barriers in feedback--ask community ambassadors, Community Relations Specialists, and research for help in solving this.
  • Work with analytics on how to measure specific populations over time as needed
  • Work with analytics on ensuring privacy through metrics.  Question if you need to keep measuring specific metrics.
  • Elevate awareness of how goals shifted over time, and why
  • Evaluate real results compared to expected
  • Have a plan for localization as you scale
  • What is community feedback?
  • Track risk factors identified in STRATEGY phase
  • Responsible and respectful exit from intensive community engagement

SUGGESTED[edit]


  • Create a “chore wheel” to gather and share user feedback
  • Throughout this process, have a “future work” document where you can collectively write down insights/learnings for the future.
  • Establish user or community councils for feedback
  • Use user feedback surveys
  • Share your DEI Prod Developments efforts publicly to get feedback
  • Identify unexpected uses
  • Identify unexpected resultsIdentify the audience actually using (and not using) it.  Identify why.

TEAM NOTES & FEEDBACK[edit]