User:MPopov (WMF)/Android/Notifications

Background
In October 2018, the Android team released an update to the Wikipedia app which added support for Notifications as part of the annual plan to improve in-app editing. Starting with version 2.7.262, users could interact with welcome, milestone, and thanks notification types inside the app and in the Android OS.

Data Collection
Using a combination of edit history in the Data Lake, EventLogging-based tracking of interactions with notifications, and data related to Echo notifications, we were able to see how engagement with notifications correlates with editing activity and new editor retention. Note: one of the ways to contribute with the mobile apps is adding Wikidata descriptions to articles without short descriptions, but for our purposes we focused specifically on article edits made on Arabic, Bangla, Chinese, Finnish, Hebrew, Hindi, Italian, Marathi, Persian, Portuguese, Russian, Swedish, and Tamil Wikipedias. (Refer to Measuring Impact for more information.)

Results
In the period studied from December 2018 through February 2019, we found that among users who have edited articles on those 13 Wikipedias with the Wikipedia Android app, 3-5% have interacted with notifications on their Android device. Newly registered editors (users who created an account during the period studied) were more likely to interact with notifications than existing editors.

Among new Android editors – which, on average, represent 29% of Android editors on those wikis – users who interacted with notifications (8%) mostly interacted with the milestone notification, with the welcome notification being a close second. Almost no users interacted with the thanks notification.

Because engagement with notifications was so low among new Android editors – i.e. 1 out of 49 on Persian Wikipedia; 4 out of 59 on Arabic Wikipedia; 7 out of 108 on Chinese Wikipedia; 3 out of 86 on Hindi Wikipedia – it is difficult to compare edit rates (average number of edits made per user within the first 30 days of user's account creation) and 7-day retention rates (% of users who made an edit in the first 7 days of their registration and then another edit in the second set of 7 days) without assumption-heavy Bayesian statistical modelling which supports extremely small sample sizes.

Of the 13 targeted Wikipedias, only Chinese Wikipedia had enough users for us to try comparing users who engaged with notifications with those who did not. As a point of reference, we also included analysis on the English Wikipedia where there were a lot more users.

Edit Rate
On Chinese Wikipedia, the average edit rate among the 7 new Android editors who interacted with notifications was 5.14 edits/user, and was 3.15 edits/user among the 101 new Android editors who did not. (This difference is not statistically significant.) For a reference point, on English Wikipedia, the average edit rate was 21.26 edits/user among 43 new Android editors who interacted with notifications, and was 5.24 edits/user among 560 new Android editors who did not. (This difference is statistically significant at the α = 0.05 level; p-value = 0.035).

Retention Rate
On Chinese Wikipedia, the 7-day retention rate among the 7 new Android editors who interacted with notifications was 20%, and was 9.9% among the 101 new Android editors who did not. (This difference is not statistically significant.) For a reference point, on English Wikipedia, the average edit rate was 15.8% among 43 new Android editors who interacted with notifications, and was 4.7% among 560 new Android editors who did not. (This difference is statistically significant at the α = 0.05 level; p-value = 0.0056).

As an interesting extra result, we also found that there were editors who were interacting with very old notifications, including ones from 2013.

Caveats
First, we had limited analytics capabilities with this feature. We could not, for example, track when a user received/has seen a notification on their device, but rather when they have interacted with one. Furthermore, our initial design for the notification preferences analytics makes it difficult to determine how many logged-in users (who are opted-in to share usage data) have enabled background polling of notifications; and our initial design does not include the interaction when a user opens the Notifications screen from the navigation menu. Both of these limitation were a result of shortsightedness on behalf of the analyst.

Second, because we did not roll this feature out as a randomized controlled trial, it is hard to infer causal relationships between notifications and edit & retention rates. If notification usage is X, a metric is Y, then there could be an unobserved confounding variable Z (e.g. user's predisposition to be an active contributor) which affects both X and Y. For example, people who use their phone more and/or have a prior history with editing wikis may be more likely to become active contributors and are more likely to engage with notifications. Although with the rate of new Android editors on these wikis, any experiment would need to run for a very long time before we would have enough data to perform any hypothesis testing with enough power.