Product Analytics/Data Products/ptwiki intervention impact report

= Executive Summary = This report is requested by product leaders to study the intervention impact on the number of edits on Portuguese Wikipedia after turning off IP editing. It concentrated on building a forecast model that would estimate the edits on Portuguese Wikipedia if IP editing was not turned off to compare with the actual edits. The model’s finding did not statistically prove turning off IP editing would negatively impact edits activeness.

= Introduction = Portuguese Wikipedia turned off IP editing on October 4th, 2020. Since then, we observed non-reverted edits (excluding bot and revert edits) decreasing in the following months. In the 20/21 fiscal year Q2, edits on Portuguese Wikipedia decreased by 0.91% year over year (Figure 1).

Considering edits on all wikipedias increased by 13.5% during the same period, we want to examine whether Portuguese Wikipedia would have seen the same increase if not turning off IP editing.

To answer this question, we provided analysis using Prophet time series forecasting method, predicted the intervention impact on edits on Portuguese Wikipedia.

= Data Characteristics = In order to estimate edits without intervention impact, we obtained monthly non-reverted edits (excluding bot and revert edits) from wmf.mediawiki_history table. The data gathered consisted of variables over a span of 69 months from July 2015 to February 2021. The variables given on a monthly basis include: edits on Portuguese Wikipedia (ptwiki), edits on all other wikipedias, and edits on all other wiki projects in Portuguese language. To explore the pattern of edits on Portuguese Wikipedia, we looked at the historical data and graphically represented it in Figure 2.

The edits almost kept flat in the last five years with a slight downward trend. The five-year trend is not purely dominated by yearly or monthly seasonality patterns. It indicated some other factors are impacting the edits. A trend only model cannot explain all the factors. We chose to use a causal model to conduct analysis and selected some wikis which are correlated with Portuguese Wikipedia as the control regressors to reflect the impact of global events in the model. After exploring 311 Wikipedias and 8 projects in Portuguese language, we selected below projects as control regressors based on correlation coefficient.


 * | Irish Wikipedia (gawiki)
 * | Russian Wikipedia (ruwiki)
 * | Sicilian Wikipedia (scnwiki)
 * | Yiddish Wikipedia (yiwiki)
 * | Portuguese Wikivoyage (ptwikivoyage)

We also explored edits by geo countries. 95% edits on Portuguese Wikipedia are from Brazil and Portugal. It has a good correlation with the edits on English Wikipedia from Brazil and Portugal. However the data is only available for a short period, not enough to forecast yearly seasonality and trend. If we have sufficient data, edits by geo countries could be a good control regressor candidate. = Model Selection = After evaluating the data and using statistical methods, we constructed multiple models for consideration.

1) model consisting of trend, seasonality, wikipedia regressors;

2) model consisting of trend, seasonality, Portuguese project regressors;

3) model consisting of trend, seasonality, Wikipedia regressors, Portuguese Wikivoyage regressor;

4) model consisting of trend, seasonality, Wikipedia regressors, Portuguese Wikivoyage regressor, pageview regressor.

We trained models using monthly data from July 2015 to September 2019 (the month before intervention), conducted 9 folds cross-validation to estimate the MAPE and evaluated the accuracy of the models (Appendix A table1, Appendix B table2). After analyzing these models, we are able to determine the 3rd model is the most effective model.

The model is structured by below three components: (Figure 3) $$Edits = EditsByTrend + EditsByYearlySeasonality + EditsByExtraRegressors $$ $$ ( gawiki, ruwiki, scnwiki, yiwiki ,ptwikivoyage ) $$

However, this model has room to improve. We discovered autocorrelation in residuals with one month lagging (Appendix B, table3). Given Prophet is a wrapped model solution, to fine tune the model we have to consider some other statistical models with more flexibility. It could be our next step to consider.

= Forecast = With the above trained model, we forecasted five months (from October 2020 to February 2021) edits without intervention. (Figure 4)

The black dots are the historical data in the pre-intervention period. Blue line and blue area are estimation and its 95% prediction interval. Red line is the actual edits after intervention. The actual number of edits is within the 95% prediction interval. In Appendix A Table1, the estimated absolute intervention impact (Actual - Prediction without intervention ) is not constantly below 0. It means there is no statistically significant evidence of edits decreasing due to intervention.

= Conclusion = As mentioned in the forecast session, we did not see the actual edits in the post-intervention period are significantly lower than the predicted edits without intervention. Therefore, we do not have statistical evidence that turning off IP editing has negatively impacted editing activeness level. Also as mentioned in the model selection session, the current model has some limitations. A fine-tuned model with other statistical methods might give a better prediction.

= Appendix A: Forecast =

= Appendix B: Model Diagnostics =

= References =