Product Analytics/Data Products/fawiki metrics summary

= Executive summary =

The Persian (Farsi) Wikipedia blocked IP editing on article pages between October 20, 2021 and April 20, 2022 by community decision. This report was commissioned by product leads to investigate the impact of the restriction on the project's health. The impacts on each metric are summarized in the below table after comparing the 6 months of restriction to the prior months in the same year, same months in prior 3 years, and other similar wiki projects. As Portuguese Wikipedia has enabled a wider restriction of IP editing on all pages, this report also looked into the similarities and differences of the impacts on the two projects.

Similar to Portuguese Wikipedia, the restriction prevented the creation of the edits that would most likely be reverted, and reduced administration page protections. Unlike Portuguese Wikipedia, the partial restriction on Persian Wikipedia did not drive new account creation or active logged-in editors, nor did it reduce the administration blocks. The impact on non-reverted content edits by users (net non-reverted non-bot content edits) is negative but not significant.

In summary, the restriction on article pages effectively reduced vandalism on Persian Wikipedia. However, the anonymous editors were not converted to registered editors after the restriction on Persian/Farsi Wikipedia. It’s recommended to re-visit the trends 6 months after the restriction was lifted, to assess whether the short-term impact is persistent or reverted.

= Introduction = The Persian/Farsi Wikipedia (fawiki) community made the decision to temporarily block anonymous editing on article (content) pages for 6 months (see consensus and ticket). Blocking was activated on October 20, 2021, Wednesday in the 42th week in 2021, and deactivated 6 months later, on April 21, 2022, Thursday in the 16th week of 2022. Per product leads’ request, the Product Analytics team provided quantitative analysis to assess the impact on the project's health. This report summarizes the trends of the metrics regarding editor, content creation and administration, and compares those metrics to the prior months, prior years, and other wiki projects.

= Methods and considerations =

Metrics selection and definition
The analysis focuses on the project metrics that are likely impacted by the restriction. Although the restriction only applied to articles, impacts on article talk pages were also assessed for possible side effects. Therefore, the page-based events such as edit, revert, protection, were evaluated on the article content pages (namespace=0) and on the article talk pages (namespace=1) separately, in addition to the overall trend on all namespaces.

Baseline selection
Three types of baselines are used in comparison analysis: (1) prior months in the same year, (2) same months in prior 3 years, and (3) other, similar wikis. This 6-month restriction was not conducted as a controlled experiment, as many variables might disturb the normal patterns and cause the changes of the metrics. For example, the pandemic is one of the biggest factors that has changed user patterns in the past two years, and its effects vary from country to country. As such, the comparison baselines should be used as a reference only instead of an assessment criterion. In the month-over-month comparison, the 6 months in restriction was compared to the period of 6 months prior to the restriction (6Mo ratio). A year of 365/366 days is grouped into the first half of the year, starting from April 21 each year, and the second half of year, starting from October 20 each year. In this division, the second half of the year usually has 1 or 2 days more than the first half of the year. Therefore, the ratio of 6 months over prior 6 months in the scenario of no impact and no seasonality is 1.005 ~1.01. To consider the seasonality, the 6Mo ratios of prior years are also referred to.

In the year-over-year comparison, the 6 months in restriction is compared to the same 6-month period of prior 3 years. 3 years prior(18/19) is pre-pandemic. 2 years prior(19/20) is the year partially in the pandemic. 1 year prior(20/21) is the year fully in the pandemic. * When IP editing was blocked on the article (content) page (namespace=0).

Based on the data from October 2019 to September 2021, 80% of edits on fawiki were submitted from Iran. In the comparison of wiki projects, six wikis were selected from the top ten wikis which were mostly edited in Iran. The six wiki projects are: Arabic Wikipedia (arwiki), Azerbaijani Wikipedia (azwiki), Central Kurdish Wikipedia (ckbwiki), Hebrew Wikipedia(hewiki), Persian Wikisource (fawikisource), Persian Wiktionary (fawiktionary). The general trend of the six wikis was assessed on whether the changes on fawiki were within a normal range. Because no two wikis are the same, the trend on the six wikis is a reference rather than a criterion. It’s observed that the Year-over-Year (YoY) or 6Month-over-6Month (6Mo6M) ratios on smaller wikis, such as Central Kurdish Wikipedia, Persian Wikisource and Persian Wiktionary tend to be more volatile than other medium size wikis. Also, the administration metrics tend to be more volatile than editors and content metrics, likely because administration actions were conducted by a small group of people for irregular needs. Four other wikis were not selected due to project-specific nuances. English Wikipedia (enwiki) was not selected because it is the biggest wiki project and could be impacted by many other global factors. South Azerbaijani Wikipedia (azbwiki) was excluded due to a poor correlation with fawiki in historical data. Turkish Wikipedia (trwiki) was not considered because trwiki had a level shift event during the comparison period – on 15 January 2020, the block of trwiki along with other language editions of Wikipedia in Turkey was lifted resulting in a metrics shift. Kurdish Wikipedia (kuwiki) was excluded because its metrics have high volatility with an unexplained spike of non-bot edits in the week of 2021-07-28, which is within the comparison period. = Impacts on editors =

Number of new accounts
Number of new accounts is defined as the number of users who register in the given period of time. The hypothesis is that the new accounts would increase when the restriction is enabled. The hypothesis is not supported.

The comparisons to prior months, prior years, and other wikis do not show an increase in new accounts on fawiki during the restriction period. Instead, Figure 1.1. Weekly Trend and Figure 1.3. YoY Comparison shows that fawiki has 18% fewer new accounts in the current year(21/22) compared to 1 year prior(20/21), dropping from 55686 to 45912. This decrease is not unique on fawiki – four other wikis have fewer new accounts in the current year(21/22)

Figure 1.2. 6Mo Comparison shows that the new accounts on fawiki in second half of year (2HY) is about same as first half of year (1HY) in the current year(21/22) while 5 out of 6 other wikis have more new accounts in second half of year than first half of year in the current year(21/22). It means that turning off IP editing on article pages did not lead to a significant increase in new accounts. It is different from the trend on Portuguese Wikipedia that new accounts surged after they blocked IP editing on the entire site. The different trends on two wikis might be due to the different blocking policies or user preference. * Restriction period

Number of active logged-in editors
Number of active user editors is defined as the number of logged-in users who made at least one content edit on fawiki in the given period of time. The hypothesis is that the active user editors would increase when the restriction is enabled. The hypothesis is not supported by data.

The comparisons to prior months, prior years, and other wikis do not show an increase in active user editors on fawiki during the restriction period. Figure 2.1. Weekly Trend shows a decline on fawiki in March and April. Figure 2.2. 6Mo Comparison shows that the active user editors on fawiki is 5% less in the second half of the current year(21/22) than the first half of year, lowest 6-month ratio in the past 4 years. 4 other wikis show a similar trend that the current year has the lowest 6M ratio. Figure 2.3. YoY Comparison shows that fawiki decreased compared to 1 year prior(20/21), but increased compared to 3 years prior(18/19). Five other wikis show a similar YoY trend. It means that turning off IP editing on article pages did not lead to a significant increase in active user editors on fawiki. It’s different from Portuguese Wikipedia that active user editors increased after the restriction was enabled on the entire site. The different trends on two wikis might be due to the different blocking policies or user preference. * Restriction period

Retention rate
Retention rate is defined as: out of the (non-bot) users who registered in the week and made at least one edit in their first 30 days, the proportion who also edited during their second 30 days.

Due to the nature of the definition, this metric surfaces two months after user account creation. It’s a metric for observing whether user retention rate will change when policy is changed. So far no obvious change in retention rate is observed(see Figure 3.1.), similar to the trend on ptwiki.

Takeaways
New accounts and active user editors did not increase on fawiki during the restriction. The trends on fawiki are different from those on ptwiki, which suggests that blocking IP editing only on article pages did not encourage anonymous editors to create accounts and edit in logged-in mode. = Impacts on content =

Number of edits
Number of edits is defined as the total number of edits made on fawiki during the given period of time. Edits that were published by bots, edits that have been reverted, and edits that have been deleted are included among total edits. This metric is for curiosity observation because it is one of the Foundation’s core product metrics. To clearly understand the impact of the restriction, the number of reverts and the number of net edits excluding bot edits, reverted edits and edits reverting other edits will be discussed in session Number of reverts and session Net non-revered edits.

Figure 4.1. Weekly Trend shows that the total edits have decreased since the restriction was enabled. Figure 4.2. 6Mo Comparison shows that total edits declined 29% during the restriction. The decrease is from content pages. The number of edits on the content page is 31% less compared to prior 6 months, contributed by the decrease in reverted edits and decrease in net non-bot non-reverted edits. The number of edits on talk pages is similar to prior 6 months. * Restriction period

Number of reverts
Number of reverts is defined as the number of edits reverted within 48 hours of being published in the given period of time. Fewer reverts is considered an improvement. The hypothesis is that the reverts on content pages would be reduced with the restriction. The hypothesis is supported by the data.

The comparisons to prior months, prior years, and other wikis show a significant reduction in content reverts on fawiki during the restriction period. Figure 6.1. Weekly Trend shows that the number of content reverts decreased when the restriction was enabled and increased when the restriction was disabled. Figure 6.2. 6Mo Comparison shows the number of content reverts is 70% less than the prior 6 months. Given in the previous 3 years, the second half of the year always has more reverts than the first half of year, it’s a significant decline. Figure 6.3. YoY Comparison shows content reverts decreased 72% compared to 1 year prior(20/21), 59% compared to 2 years prior(19/20), 45% compared to 3 years prior(18/19). Other wikis do not show such a decline. This trend only shows on fawiki.

It means restriction prevents creating content which most likely will be reverted.

Reverts on the talk page only account for 1% of the total reverts. Figure 7.1 Weekly Trend shows that the number of talk reverts does not have a significant change. As usual, the number of talk reverts is slightly higher in the second half of the year than in the first half of the year. (see Figure 7.2.)

* Restriction period

Net non-reverted edits
* Restriction period

* Restriction period

Takeaways
= Impacts on administration =

Protected pages


* Restriction period

Takeaways
= Takeaways =

= References =