Wikimedia Apps/iOS Suggested edits project/Alt Text Experiment

Background

In fiscal year 2023-2024, the iOS team worked to increase unreverted mobile contributions. One of our hypotheses to achieve this growth was to introduce Suggested Edits to the iOS app. We first explored a suggested edits task that encouraged users to add helpful and accurate alt-text to the 95% of images across Wikipedias that are missing them. We created an alt text proof-of-concept experimental build, and shared it with users at the 2023 GLAM conference.

After getting feedback, we feel confident with scaling our experiment in the production version of app. Part of this experiment builds on top of our recent addition to the iOS app of the "Add an image" suggested edit.

This work is part of the 2024-2025 annual plan Wiki Experiences 1, focused on Contributor Experience.

Our overall hypothesis is: If we conduct an A/B/C test with the alt-text suggested edits prototype in the production version of the iOS app, we can determine if adding alt-text to images is a good task for newcomers or should be reserved for experienced users.

Experiment Design

Research questions

• Does reminding users to add alt text prove an effective way to increase the number of edits that include alt text?
• Should adding Alt text for images in articles be a stand-alone suggested edit? (And if so, would it be an appropriate task for newcomers?)
• Would users appreciate prompts for other improvements they can make to the article they are editing?
• Does providing alt text guidance during the flow create higher quality alt-text?

Experiment B: Add alt text Prompt after "Add an Image" flow

Currently, only around 8% of editors completing Image Recommendations add alt text. We want to learn if reminding users to add alt text prove an effective way to increase the number of edits that include alt text. Editors will only be shown the prompt once.

• Hypothesis: if we prompt users once to add missing alt text to their image after they have published an image using the Image Recommendations suggested edit, we'll see 60% of editors choose to additionally publish alt text, and 10% will add alt-text for subsequent image recommendation edits made in the next 15 days.
• Group A = 50% of "Add an image" users, control group
• Group B = 50% of "Add an image" users, shown prompt to add alt text after their edit is published if alt text = Null.

Experiment C: Add alt text Prompt after standard editing flow

We want to learn if users would appreciate prompts for other improvements they can make to the article they are editing. This is a type of in-time suggested edit. Editors will only be shown the prompt once.

• Hypothesis: if we prompt users to add missing alt text after they have published an edit on any article containing an image in need of alt text, 4% of editors will go on to add alt text to the image.
• Group D = 50% of logged-in editors editing an article with an image, control group
• Group C = 50% of all logged-in editors receive encouragement once to add alt-text to an image when editing an article with an image

Feature Requirements

Must haves

• Entry point will vary based on experiment group (see below)
• If a user answers "no" to "Add alt text" prompt, gain reason for why
• Prominent guidance for writing good alt-text
• Users ability to get context about the image from the article
• Users ability to access relevant metadata (can take user to Web)
• Detection of which images do not have alt-text
• Ability to publish alt-text
• Alt-text edit should count as a separate edit ONLY when submitted separately
• Alt text is published with automatic edit summary
• Users should be prompted provide feedback about the feature
• Instrumentation that allows us to evaluate the alt-text submitted
• Input field for alt-text in context of image, preview of alt-text in context of our existing editor
• Warning when alt-text exceeds 125 characters

Nice to have

• Positive Reinforcement
• Do not allow users to copy and paste in image caption
• Suggest Alt-text (think Machine Assisted Article Descriptions)
• Can playback what was written in preview
• Surface Categories and Depicts

Experiment B Must Haves Experiment Group B (50% of Image Recs Suggested Edits users). They will enter the dedicated flow for Alt Text AFTER adding an image to an article using Image Recommendations.

Entry point: After image recommendations edit is submitted; if alt-text = null:

1. Educate user on importance of alt-text and ask if they’d be willing to add alt-text to image
2. If user selects yes; launch alt-text adding flow for most recently added image

After task completion:

1. Show a survey when to ask users if it should be a separate task
2. Return users to next image recommendations suggestion

Experiment C Must Haves Experiment Group C: 50% of all editors receive encouragement to add alt-text to an image when editing an article with an image

Entry point:

• After edit where either of the following were true

1. No edit was made to an image, but there is an image in the article without alt text
2. Edit made to image but alt-text = null

• Show prompt that an image in the article is in need of alt-text and explain importance of adding alt-text and ask users if they’re willing to add alt-text
• If users selects Yes, launch alt-text adding flow for one image in article in need of alt text

After task completion

1. Ask satisfaction
2. Ask if this should be a dedicated task
3. Bring user back to article they were reading

Experiment C Nice to Haves

• Affordance to reveal / review alt-text on images in an article
• If someone is abandoning an edit, launch the prompt for adding an image if they have over 50 edits

Target Wikis

While we welcome feedback from everyone, we are especially interested in hearing from:

• Spanish, French and Portuguese speakers in the Americas (North, South and Central) and Caribbean
• Chinese speakers in North America

We are planning to run this experiment in partnership with the following wikis:

• Spanish Wikipedia
• French Wikipedia
• Portuguese Wikipedia
• Chinese Wikipedia

How will we know we are successful?

Evaluating the quality of Alt Text:

In our preliminary research, we learned that alt text additions are not formally patrolled, so we may not be able to rely on revert rate alone to understand the quality of alt text being added. We plan to partner with an accessibility organization to evaluate the quality of the alt text produced through this experiment. We will ask them to rate each alt text entry from 1-5 with 5 being the highest quality, and 1 being lowest quality. They will also provide a predicted revert score, answering the question: if they saw that alt text added to an image, would they remove it from Wikipedia?

Key Indicators

1. 5% increase in the percentage of newcomers who publish ≥1 constructive edit in the main namespace on the iOS App during the experiment
2. 60% of experiment group B editors publish an additional edit with alt text for the image they were prompted on
3. 10% of experiment group B editors add alt-text for subsequent image recommendation edits made in the next 15 days
4. Alt text added through our prompts have a 50% higher rating score on average, than those made without guidance. (Group B & C average alt text scores compared with Group A's average alt text scores)
5. 4% of experiment group C users add alt text when prompted after editing an article
6. If we do not get 200 edits to evaluate for alt text quality, from at least 50 unique editors, we will add more languages to the experiment audience
7. If 71% of edits are scored a 3 or higher we will scale the feature. If less than 70% of edits are scored a 3 or higher we will improve guidance or use AI to better assist users.
8. If quality scores for Newcomers (users with < 50 edits) are more than 50% worse than quality scores for experienced editors (50+ edits), we will not recommend this task be available to newcomers.
9. If we see at least 60% of users would like to see it as a separate task, then we will have the confidence to pursue a feed of alt-text suggested edits (Group B + Possibly C if secondary survey is added)
10. If 60% or more of respondents say they would be interested in similar edit notifications for articles they are working on (Group C survey responses), we share this information and consider future edit prompts.

Guardrails

1. Retention rate of editors in group B or C who have received an Alt text prompt does not differ from controls by more than 10%
2. Revert rate for alt-text edits from experiment groups does not exceed control by more than 5 percentage points
3. Revert rate for Newcomers does not exceed 18%
4. Edit over acceptance rate for Newcomers is not more than 5 percentage points lower than Growth’s comparable "Add an Image" edit over acceptance rate

Curiosities

1. Are those who received the prompt more likely to make alt-text edits in the following 15 days? (Group C non-image edits before prompt, vs after prompt)
2. How does the predicted revert rate compare to Android's Image Captions Suggested Edit?
3. Is there a difference in metrics by language and geography?(For example, breaking down edits from Latin America vs Europe for Spanish)

Designs

Below are the initial designs for the experiments from April 2024:

• 
  Add missing alt text modal
• 
  Onboarding tips (5 points)
• 
  First tooltip: Review article
• 
  Second tooltip: Inspect image
• 
  Third tooltip: Add description
• 
  Initial view to add alt text
• 
  Info-i icon leads to image information from common's listing
• 
  Guidance for writing alt text
• 
  Inputting alt text
• 
  Preview screen with alt text on image
• 
  Automatic edit summary is added, confirmation after publishing

How to follow along

We have created T357437, Alt-Text Suggested Edit Scaled Experiment on iOS as our Phabricator epic. We encourage your collaboration there or on our Talk Page. There will also be periodic updates to this page as we make progress on the experiment.

Updates

September 2023 - April 2024

For earlier updates about the Alt Text suggested edit, see the overview page's updates.