Wikimedia Apps/Team/Android/Image Recommendations
Background[edit]
In May 2021 the Android team built an minimum viable product of Image recommendations. The goal of the MVP was to determine if it would be a good task for newcomers, especially on mobile. The results were generally positive resulting in the Growth team introducing the feature on Web (Desktop and Mobile) on select language wikis. The Structured Data team also released a version of an image suggesting tool targeted at experienced users across all Wikis on web via notifications. Due to both tools having generally positive results, and to meet requests of bringing the feature back to the app, we are re-releasing Image Recommendations. Instead of the tool validating the accuracy of the suggestion like the original MVP, this version will enable experienced editors to place suggested images into Wikipedia articles.
Feature Requirements[edit]
Essentials for Version 1[edit]
The entry point for users should be located both on the 'Suggested Edits' screen and within an article.
- 'Image Recommendations' task should be positioned immediately below 'Article Descriptions'.
- A new task indicator should be available for users who have already completed at least one suggested edit. This indicator should disappear upon their second visit.
- Users should be able to add captions and alt text
- Users must have an accessible avenue to report feature-related issues, which will lead them to our support email.
- A link to the FAQ page should be included within the task.
- Measures should be in place to prevent users from repeatedly confirming edits within short intervals (less than 5 seconds) to guard against misuse.
- The first time a user opens the task they will be guided through an onboarding process for the feature.
- Users should have the ability to access the full article and image metadata.
- Users cannot skip tasks without providing some form of feedback (Yes, No, Not Sure).
- Users must be able to zoom/pinch on the image
- Provisions need to be made for scenarios when images run out of tasks or tasks are unavailable in certain languages.
- Suggested Edit should not be available to users with less than 50 edits
Nice to Haves[edit]
- Enabling users to filter tasks by topics. Incorporation of subtle positive reinforcement messages (displayed in dark mode).
Out of scope[edit]
- Addition of section images.
Target Wikis[edit]
While we welcome feedback from everyone, we are especially interested in hearing from:
- Spanish and Portuguese Wikipedia editors within Latin American and Caribbean countries.
- Hindi Wikipedia editors residing in India.
- Persian Wikipedia editors distributed across the diaspora.
In our study, we are committed to fostering a balanced and diverse group of testers. To that end, we aim for a broad spectrum of gender and age representation, ensuring that all perspectives are well-captured and accounted for.
User Stories[edit]
- As a Wikipedia Android app user with a small screen and inconsistent internet, I would like to evaluate images and determine if they should go into an article, to contribute to Wikipedia articles that are in need of more content.
- When I am using the Android app, I want to be able to add images with one hand to many articles on my mobile device, so that I can be productive while riding the bus in BogotĂ , and listening to music.
- When I am reading an article about Allameh Jafari Bridge, I want to learn that I can add an image to the Persian language article, so that I can enhance articles that I find interesting.
- When I try to add an image to an article and fail, I want to learn how to do this task, so that I can build my confidence before going on to adding image that I want as my intended edit.
- BONUS: When I am recommended an image for an article in Bengali and I donât believe the image is a good fit and I am physically next to the subject of the article, I want to be made aware there is a Commons app, so I can upload the appropriate image using the Commons App
How will we know we are successful?[edit]
Validation[edit]
- KR 1.1: 2000 articles have images in a 30 day period
- KR 1.2: Average at least 8 edits per session per unique user
- KR 1.3: 15% of Suggested Editors try image recommendations task
- KR 1.4: 70% of those users complete the task again on a separate day in a 15 day period
- KR 1.5: Reject and Accept rate does not deviate from Mobile Web or MVP by more than 10%
- KR 1.6: DAU of Suggested Edits increase overall
Guardrails[edit]
- KR 1.1: Feature does not worsen gender or geographic bias
- KR 1.2: Less than 5% of users report NSFW or offensive content
- KR 1.3: Users spend at least 10s evaluating a task before publishing it
- KR 1.4: Bounce rate does not exceed 50%
- KR 1.5: At least a 35% task completion rate
- KR 1.6: Revert rate does not exceed 18%
Curiosities[edit]
- KR 1.1: Do these numbers differ by language or user tenure
- KR 1.2: If this is a userâs first suggested edit, do they go on to try others?
- KR 1.3: Feature perception by geographically underrepresented groups on large language wiki
- KR 1.4 At what point in workflow are most frequent dropoff events?
Designs[edit]
Below are initial designs for the feature:
-
Discovery of Image Recommendations -
Image Recommendations -
Onboarding to Image Recommendations -
Contextual tooltips onboard the user to the task -
Contextual tooltips onboard the user to Image Recommendations 2 -
Contextual tooltips onboard the user to Image Recommendations 3 -
Image Recommendations Feed -
Image Recommendations Rejection dialog -
Image Recommendations Overflow menu -
Image Recommendations Sheet Collapsed -
Image Recommendations Image Details -
Image Recommendations File Page -
Image Recommendations Image Detail Active -
Image Recommendations Image Detail Filled -
Image Recommendations Preview -
Edit summary -
Edit summary active -
Edit summary loading
Updates[edit]
January 2024- Results after 30 days[edit]
Added the capability to copy and paste text within the image editing panel so users can reuse text between captions and other fields as long as text is under 50 characters.T335644
The Add an image suggested edit task has been available to users in the Android App as of September 2023. The findings from our 30-day Analysis for the period of to can be found below. Results have been positive overall, and we are currently working to implement this task in the iOS app as well.
Validation[edit]
- KR 1.1: Goal: 2000 articles have images in a 30 day period
- Actual: 2118 articles had added images across 25 Wikis in a 30 day period
| Language | Count of Edits |
|---|---|
| English | 1124 |
| German | 361 |
| Russian | 113 |
| French | 102 |
| Italian | 75 |
| Turkish | 62 |
| Chinese | 54 |
| Portuguese | 51 |
| Persian | 26 |
| Japanese | 24 |
- KR 1.2: Goal: Average at least 8 edits per session per unique user
- Actual: The average daily edit/editor was 6.5 average edits per editor in 30 day period. Overall the average is 13.8. The MVP had a higher edit per session per unique user rate, and we believe this is due to the positive reinforcement that was included in the MVP.
- KR 1.3: Goal: 15% of Suggested Editors try image recommendations task
- Actual: 12.7% of eligible Suggested Editors tried image recommendations task.
- KR 1.4: Goal: 70% of those users complete the task again on a separate day in a 15 day period
- Actual: 34.5% of those users completed the task again on a separate day in a 15 day period. 18.1% returned on 3 or more days. Although we did not meet our goal, this retention rate is quite high for mobile editing.
- KR 1.5: Goal: Reject and Accept rate does not deviate from Mobile Web or MVP by more than 10 percentage points.
- Actual: Accept rate does not deviate from Mobile Web or MVP by more than 10 percentage points, however Android Reject Rate was significantly lower at 8.8% vs Growthâs average 44.2% Reject Rate. The Android Skip Rate is much higher than Growth's which may be attributed to the difference in language used (Skip vs Not Sure). It could also be a difference in audience, this task was for people with over 50 unreverted edits, whereas the Growth task is for newcomers. The MVP did not publish the image so users may have been more confident to accept the image.
| Metric | Android | Growth Add an Image Mobile Web | Android MVP |
|---|---|---|---|
| Image Suggestion Accept Rate | 33.1% | 38.8% | 64% |
| Reject Rate | 8.8% | 28.1% | 20% |
| Skip/Not Sure Rate | 55.0% | 12.7% | 14% |
- KR 1.6: Daily Active Users (DAU) of Suggested Edits increase overall
- Actual: Editor DAU increased by 4.8% compared to September 2023.
Guardrails[edit]
- KR 1.1: Goal: Feature does not worsen gender or geographic bias
- Analysis forthcoming
- KR 1.2: Goal: Less than 5% of users report NSFW or offensive content
- Actual: 0.6% of users reported NSFW or offensive content. Of 496 Rejection Reasons submitted, 1 was category 'Offensive', 0.2% percent of Reports
- KR 1.3: Goal: Users spend at least 10s evaluating a task before publishing it
- Actual: Average time spent on the task was 150.8 seconds, and the median time spent was 68.7 seconds. There were no edits made in under 10 seconds.
- KR 1.4: Goal: Bounce rate does not exceed 50%
- Actual: Bounce rate was 28.7% (Bounce rate defined as users that click begin the task, and then abandon the flow before publishing their edit)
- KR 1.5: Goal: At least a 35% task completion rate
- Actual: 99.4% Task Completion rate (Defined as users that click on Add an image as a task, and then clicks Yes, No or Skip/Not sure (interact with the feature)
- KR 1.6: Goal: Revert rate does not exceed 18%
- Actual: Revert rate did not exceed 3.8%, slightly higher than other suggested edits, and lower than Growth Add an Image task.
| Edit Type | Revert rate |
|---|---|
| Android Add Image Suggested Edits | 3.8% |
| Other Android Suggested Edits | 2.2% |
| Growth Add an Image Mobile Web | 13.10% |
| Growth Add an Image Desktop | 8.30% |
Curiosities[edit]
- KR 1.1: Do these numbers differ by language or user tenure
- Actual: Broken out by Wiki, Revert Rate ranges between 27.3% (Bosnian) and 3.1% (English). It is noteworthy Bosnian Wikipedia only had 11 Image Recommendations edits at the time of this analysis.
- KR 1.4: At what point in workflow are most frequent drop-off events?
- Actual: Caption Entry has the highest drop off rate (task steps not completed) at 32.6%
| Funnel Step | Bounce Rate |
|---|---|
| Image Decision Task | 0.6% |
| Caption Entry Task | 32.6% |
| Edit Summary Task | 2.1% |
- KR 1.5: How often are users adding caption to images?
- Actual: 73.5% of edits were captioned.
- KR 1.5b: Learn: What was the revert reason, was it because of lack of caption? How often are reverted edits captioned?
- Actual: No direct relationship was demonstrated between reverts and captioned image status.
November 2023[edit]
- After receiving feedback from German Wikipedia, we updated the restriction for seeing image recommendations on German Wikipedia to a minimum of 50 unreverted edits on German Wikipedia, instead of global edits. (T351275)
October 2023[edit]
- We did a 15 day check on our data to ensure the feature is on the right track. Below our the results of our 15 day check:
| Goal | Actuals |
|---|---|
| Image Rejection Rate <29% | 11.4% |
| Edit Over Acceptance Rate <35% | 8.60% |
| Task Completion Rate is not Below 30% | 60.8% |
| Revert Rate <18% | 1.60% |
Although 992 edits were completed during this period; the chart is based on 456 edits, with the exception of revert rate.
- We encourage all editors with over 50 edits to give it a try and provide feedback via the overflow menu in the feature.
- We have begun work on the Places (also known as Nearby) on Android feature. Updates about the feature can be found on the project page.
September 2023[edit]
Image Recommendations on Android is available in the Beta version of the app for users that over 50 edits. The feature can be accessed by experienced logged-in users by going to the edits tab in the main navigation bar.
August 2023[edit]
We expect this feature to be released by the end of September. We are currently code complete, however there are a few bugs that other teams need to work out before we can release.
June 2023[edit]
API Underpinnings and modifications to the Growth extension to enable the feature in the app was made available. We also completed the designs found in the section above.