User:MPopov (WMF)/Android/Notifications

Background
In October 2018, the Android team released an update to the Wikipedia app which added support for Notifications as part of the annual plan to improve in-app editing. Starting with version 2.7.262, users could interact with welcome, milestone, and thanks notification types inside the app and in the Android OS:

Impact Goals
With this release we were targeting:
 * 10% increase in 7-day retention rate
 * a new editor is 7-day-retained if they make an edit (either article edit or a Wikidata description edit) in first 7 days after creating account and then edit again in the second set of 7 days
 * 10% increase in edit rate (edits per user)
 * the total number of article edits and Wikidata description edits made specifically with the Wikipedia Android app in the user's first 30 days after creating their account

Data Collection
Using a combination of edit history in the Data Lake, EventLogging-based tracking of interactions with notifications, and data related to Echo notifications, we were able to see how engagement with notifications correlates with editing activity and new editor retention. We focused specifically on article edits made on the following Wikipedias:
 * Arabic
 * Bangla
 * Chinese
 * Finnish
 * Hebrew
 * Hindi
 * Italian
 * Marathi
 * Persian
 * Portuguese
 * Russian
 * Swedish
 * Tamil

and Wikidata description edits made in those languages. Refer to Measuring Impact for more information on how these target wikis were chosen.

Results
We compared edit rate and retention rate among new Android editors between July 2018 and February 2019. Both Wikipedia article edits and Wikidata description edits counted towards these two metrics. Since we relied on the February 2019 snapshot of editing history in the Data Lake, we employed the following conditions:


 * For the edit rate – since it relies on a 30-day window after the user's registration – we excluded users who registered their account after 29 January 2019.
 * For the retention rate – since it relies on a 14-day window after the user's registration – we excluded users who registered their account after 14 February 2019.

The before/after values are presented in the table below and the figure to the right. Among these target languages:


 * The average 30-day edit rate increased by 92% (from an average of 2.69 edits/user to an average of 5.17 edits/user)
 * The average 7-day retention rate increased by 69.7% (from an average of 6.1% to an average of 10.3%)

As an interesting extra result, we also found that there were editors who were interacting with very old notifications, including ones from 2013.

Caveats
First, because we did not roll this feature out as a randomized controlled trial, it is hard to infer causal relationships between notifications and edit & retention rates. If notification usage is X, a metric is Y, then there could be an unobserved confounding variable Z (e.g. user's predisposition to be an active contributor) which affects both X and Y. For example, people who use their phone more and/or have a prior history with editing wikis may be more likely to become active contributors and are more likely to use notifications. Although with the rate of new Android editors on these wikis and low engagement with notifications, any experiment would need to run for a very long time before we would have enough data to perform any hypothesis testing with enough power. Furthermore, the release of notifications was combined with the release of the navigation update, although a change to the navigation UX would probably not have an effect on editing activity.

Second, we had limited analytics capabilities with this feature. We could not, for example, track when a user received/has seen a notification on their device, but rather when they have interacted with one. Furthermore, our initial design for the notification preferences analytics makes it difficult to determine how many logged-in users (who are also opted-in to share usage data) have enabled background polling of notifications; and our initial design does not include the interaction when a user opens the Notifications screen from the navigation menu.