Product Analytics/Data Products/fawiki metrics summary

= Executive summary =

The Persian (Farsi) Wikipedia blocked IP editing on article pages between October 20, 2021 and April 20, 2022 by community decision. This report was commissioned by product leads to investigate the impact of the restriction on the project's health. The impacts on each metric are summarized in the below table after comparing the 6 months of restriction to the prior months in the same year, same months in prior 3 years, and other similar wiki projects. As Portuguese Wikipedia has enabled a wider restriction of IP editing on all pages, this report also looked into the similarities and differences of the impacts on the two projects.

Similar to Portuguese Wikipedia, the restriction prevented the creation of the edits that would most likely be reverted, and reduced administration page protections. Unlike Portuguese Wikipedia, the partial restriction on Persian Wikipedia did not drive new account creation or active logged-in editors, nor did it reduce the administration blocks. The impact on non-reverted content edits by users (net non-reverted non-bot content edits) is negative but not significant.

In summary, the restriction on article pages effectively reduced vandalism on Persian Wikipedia. However, the anonymous editors were not converted to registered editors after the restriction on Persian/Farsi Wikipedia. It’s recommended to re-visit the trends 6 months after the restriction was lifted, to assess whether the short-term impact is persistent or reverted.

= Introduction = The Persian/Farsi Wikipedia (fawiki) community made the decision to temporarily block anonymous editing on article (content) pages for 6 months (see consensus and ticket). Blocking was activated on October 20, 2021, Wednesday in the 42th week in 2021, and deactivated 6 months later, on April 21, 2022, Thursday in the 16th week of 2022. Per product leads’ request, the Product Analytics team provided quantitative analysis to assess the impact on the project's health. This report summarizes the trends of the metrics regarding editor, content creation and administration, and compares those metrics to the prior months, prior years, and other wiki projects.

= Methods and considerations =

Metrics selection and definition
The analysis focuses on the project metrics that are likely impacted by the restriction. Although the restriction only applied to articles, impacts on article talk pages were also assessed for possible side effects. Therefore, the page-based events such as edit, revert, protection, were evaluated on the article content pages (namespace=0) and on the article talk pages (namespace=1) separately, in addition to the overall trend on all namespaces.

Baseline selection
Three types of baselines are used in comparison analysis: (1) prior months in the same year, (2) same months in prior 3 years, and (3) other, similar wikis. This 6-month restriction was not conducted as a controlled experiment, as many variables might disturb the normal patterns and cause the changes of the metrics. For example, the pandemic is one of the biggest factors that has changed user patterns in the past two years, and its effects vary from country to country. As such, the comparison baselines should be used as a reference only instead of an assessment criterion. In the month-over-month comparison, the 6 months in restriction was compared to the period of 6 months prior to the restriction (6Mo ratio). A year of 365/366 days is grouped into the first half of the year, starting from April 21 each year, and the second half of year, starting from October 20 each year. In this division, the second half of the year usually has 1 or 2 days more than the first half of the year. Therefore, the ratio of 6 months over prior 6 months in the scenario of no impact and no seasonality is 1.005 ~1.01. To consider the seasonality, the 6Mo ratios of prior years are also referred to.

In the year-over-year comparison, the 6 months in restriction is compared to the same 6-month period of prior 3 years. 3 years prior(18/19) is pre-pandemic. 2 years prior(19/20) is the year partially in the pandemic. 1 year prior(20/21) is the year fully in the pandemic. * When IP editing was blocked on the article (content) page (namespace=0).

Based on the data from October 2019 to September 2021, 80% of edits on fawiki were submitted from Iran. In the comparison of wiki projects, six wikis were selected from the top ten wikis which were mostly edited in Iran. The six wiki projects are: Arabic Wikipedia (arwiki), Azerbaijani Wikipedia (azwiki), Central Kurdish Wikipedia (ckbwiki), Hebrew Wikipedia(hewiki), Persian Wikisource (fawikisource), Persian Wiktionary (fawiktionary). The general trend of the six wikis was assessed on whether the changes on fawiki were within a normal range. Because no two wikis are the same, the trend on the six wikis is a reference rather than a criterion. It’s observed that the Year-over-Year (YoY) or 6Month-over-6Month (6Mo6M) ratios on smaller wikis, such as Central Kurdish Wikipedia, Persian Wikisource and Persian Wiktionary tend to be more volatile than other medium size wikis. Also, the administration metrics tend to be more volatile than editors and content metrics, likely because administration actions were conducted by a small group of people for irregular needs. Four other wikis were not selected due to project-specific nuances. English Wikipedia (enwiki) was not selected because it is the biggest wiki project and could be impacted by many other global factors. South Azerbaijani Wikipedia (azbwiki) was excluded due to a poor correlation with fawiki in historical data. Turkish Wikipedia (trwiki) was not considered because trwiki had a level shift event during the comparison period – on 15 January 2020, the block of trwiki along with other language editions of Wikipedia in Turkey was lifted resulting in a metrics shift. Kurdish Wikipedia (kuwiki) was excluded because its metrics have high volatility with an unexplained spike of non-bot edits in the week of 2021-07-28, which is within the comparison period. = Impacts on editors =

Number of new accounts
Number of new accounts is defined as the number of users who register in the given period of time. The hypothesis is that the new accounts would increase when the restriction is enabled. The hypothesis is not supported.

The comparisons to prior months, prior years, and other wikis do not show an increase in new accounts on fawiki during the restriction period. Instead, Figure 1.1. Weekly Trend and Figure 1.3. YoY Comparison shows that fawiki has 18% fewer new accounts in the current year(21/22) compared to 1 year prior(20/21), dropping from 55686 to 45912. This decrease is not unique on fawiki – four other wikis have fewer new accounts in the current year(21/22)

Figure 1.2. 6Mo Comparison shows that the new accounts on fawiki in second half of year (2HY) is about same as first half of year (1HY) in the current year(21/22) while 5 out of 6 other wikis have more new accounts in second half of year than first half of year in the current year(21/22). It means that turning off IP editing on article pages did not lead to a significant increase in new accounts. It is different from the trend on Portuguese Wikipedia that new accounts surged after they blocked IP editing on the entire site. The different trends on two wikis might be due to the different blocking policies or user preference. * Restriction period

Number of active logged-in editors
Number of active user editors is defined as the number of logged-in users who made at least one content edit on fawiki in the given period of time. The hypothesis is that the active user editors would increase when the restriction is enabled. The hypothesis is not supported by data.

The comparisons to prior months, prior years, and other wikis do not show an increase in active user editors on fawiki during the restriction period. Figure 2.1. Weekly Trend shows a decline on fawiki in March and April. Figure 2.2. 6Mo Comparison shows that the active user editors on fawiki is 5% less in the second half of the current year(21/22) than the first half of year, lowest 6-month ratio in the past 4 years. 4 other wikis show a similar trend that the current year has the lowest 6M ratio. Figure 2.3. YoY Comparison shows that fawiki decreased compared to 1 year prior(20/21), but increased compared to 3 years prior(18/19). Five other wikis show a similar YoY trend. It means that turning off IP editing on article pages did not lead to a significant increase in active user editors on fawiki. It’s different from Portuguese Wikipedia that active user editors increased after the restriction was enabled on the entire site. The different trends on two wikis might be due to the different blocking policies or user preference. * Restriction period

Retention rate
Retention rate is defined as: out of the (non-bot) users who registered in the week and made at least one edit in their first 30 days, the proportion who also edited during their second 30 days.

Due to the nature of the definition, this metric surfaces two months after user account creation. It’s a metric for observing whether user retention rate will change when policy is changed. So far no obvious change in retention rate is observed(see Figure 3.1.), similar to the trend on ptwiki.

Takeaways
New accounts and active user editors did not increase on fawiki during the restriction. The trends on fawiki are different from those on ptwiki, which suggests that blocking IP editing only on article pages did not encourage anonymous editors to create accounts and edit in logged-in mode. = Impacts on content =

Number of edits
Number of edits is defined as the total number of edits made on fawiki during the given period of time. Edits that were published by bots, edits that have been reverted, and edits that have been deleted are included among total edits. This metric is for curiosity observation because it is one of the Foundation’s core product metrics. To clearly understand the impact of the restriction, the number of reverts and the number of net edits excluding bot edits, reverted edits and edits reverting other edits will be discussed in session Number of reverts and session Net non-revered edits.

Figure 4.1. Weekly Trend shows that the total edits have decreased since the restriction was enabled. Figure 4.2. 6Mo Comparison shows that total edits declined 29% during the restriction. The decrease is from content pages. The number of edits on the content page is 31% less compared to prior 6 months, contributed by the decrease in reverted edits and decrease in net non-bot non-reverted edits. The number of edits on talk pages is similar to prior 6 months. * Restriction period

Number of reverts
Number of reverts is defined as the number of edits reverted within 48 hours of being published in the given period of time. Fewer reverts is considered an improvement. The hypothesis is that the reverts on content pages would be reduced with the restriction. The hypothesis is supported by the data.

The comparisons to prior months, prior years, and other wikis show a significant reduction in content reverts on fawiki during the restriction period. Figure 6.1. Weekly Trend shows that the number of content reverts decreased when the restriction was enabled and increased when the restriction was disabled. Figure 6.2. 6Mo Comparison shows the number of content reverts is 70% less than the prior 6 months. Given in the previous 3 years, the second half of the year always has more reverts than the first half of year, it’s a significant decline. Figure 6.3. YoY Comparison shows content reverts decreased 72% compared to 1 year prior(20/21), 59% compared to 2 years prior(19/20), 45% compared to 3 years prior(18/19). Other wikis do not show such a decline. This trend only shows on fawiki.

It means restriction prevents creating content which most likely will be reverted.

Reverts on the talk page only account for 1% of the total reverts. Figure 7.1 Weekly Trend shows that the number of talk reverts does not have a significant change. As usual, the number of talk reverts is slightly higher in the second half of the year than in the first half of the year. (see Figure 7.2.)

* Restriction period

Net non-reverted edits
The net non-reverted edits is defined as the number of edits which were not reverted within 48 hours, excluding bot edits, reverted edits and edits reverting other edits. They are considered as ‘good’ edits by users.

The first hypothesis is that the net non-reverted edits on all namespaces would decline. It’s inconclusive. Figure 8.1. Weekly Trend shows that the net non-reverted edits dropped since October 20,2021. There was a spike in the week of 2022-03-21, 18221 edits created by one editor within a week (mostly in 2 days). Even though the user is not identified as a bot, their behavior is similar to a bot. So the number of net edits in the second half of the current year(21/22) was adjusted by excluding the 18221 edits. Figure 8.2. 6Mo Comparison shows that after the adjustment net non-reverted edits on fawiki is 17% less in second half of year than in the first half of year of the current year(21/22), dropping from 917617 in first half of year to 759026 in second half of year. It is a common trend on 4 other wikis in the current year(21/22) that the second half of the year has less net edits. Figure 8.3. YoY comparison shows that the current year(21/22) is 18% lower than the 1 year prior(20/21), but still 6% higher than 2 years prior(19/20) and 13% higher than 3 years prior(18/19). In summary, though a drop right after the restriction was confirmed, it’s inconclusive that the overall net edits decreased during the six months of restriction given other wikis had declined during the same period of time and that current edits level is higher than prior 2 years prior(19/20) and 3 years prior(18/19).

The second hypothesis is that the net non-reverted edits on content pages would decline. It is inconclusive. Net edits on content pages contributed 70% of total net edits. The trend on content pages is similar to the trend of net edit on all namespaces. (see Figure 9.1) The net content edits decreased once the restriction was enabled in Oct 2022. The net non-reverted edits on content pages is 24% less in second half of the current year(21/22) than in the first half of the year, dropping from 657228 to 498571.(see Figure 9.2) However it’s inconclusive that the overall net content edits decreased during the restriction given that other wikis had declined during the same period of time and that current edits level is higher than prior 2 years prior(19/20) and 3 years prior(18/19).(see Figure 9.3) If assume the organic 6-month ratio in the current year(21/22) on fawiki without restriction is 0.925, the median of the other 6 wikis, it is estimated that fawiki lost additional 109,365 content edits during restriction, i.e. 17% of total net content edits in first half of year of the current year(21/22).

The third hypothesis is that the net non-reverted edits on the talk page would increase. It is not supported. Figure 10.1. Weekly Trend shows the level of net talk edits is close to the previous level. Figure 10.2. 6Mo Comparison shows the net talk edits in the second half of year is 14% higher than the first half of year in the current year(21/22). It is still within the historical range. Figure 10.3. YoY Comparison shows the net talk edits in the current year (21/22) is 13% less than 1 year prior(20/21) and 1% less than 2 years prior(19/20), 6% more than 3 years prior(18/19). In summary, net talk edits did not increase significantly. * Restriction period

* Restriction period

Takeaways
Restriction prevents content creation which most likely will be reverted. Restriction might prevent creating good content at some level. But the effect is not significant enough to be confirmed. = Impacts on administration =

Number of blocks
Number of blocks is defined as the number of block and reblock actions on accounts or IPs or IP ranges in a given period of time. Fewer blocks is considered an improvement. The hypothesis is that blocks would be reduced when restriction is enabled. This hypothesis is not supported by the data.

Figure 11.1. Weekly Trend shows that the number of blocks reduced in October and November, 2021, but spiked twice in February 2022 and March 2022. It’s different from ptwiki which showed a significant decrease in blocks.

Protected pages
Number of protected pages is defined as the number of unique pages protected in a given period of time. Having fewer protected pages is considered an improvement. The hypothesis is that the number of protected pages would reduce during restriction. This hypothesis is supported.

Figure 12.1 Weekly Trend shows that the metric spiked in October 2021. 5867 pages were protected by one user. Per a community member’s request, those protections were excluded from the total of protected pages in the first half of the current year(21/22). After the adjustment, compared to the prior 6 months, the number of protected pages decreased 32%. The reduction in protected pages is from content pages, as the protections on talk pages only account for 1% of the total.

* Restriction period

Checkuser checks
Number of Checkuser checks is defined as the number of checkuser requests in a given period of time. The metric is for curiosity observation. Figure 13.1 Weekly Trend shows the number of checkuser checks has no significant change, staying close to its previous level.

Takeaways
Restriction does not impact the administration actions significantly. Restriction on content pages reduced page protections but did not show impact on blocks and checkuser requests.

= Takeaways =


 * The restriction on Persian Wikipedia article pages prevented content creation which would likely be reverted. But the non reverted edits slightly declined.
 * The restriction on Persian Wikipedia article pages did not drive new editors and active editors.
 * The restriction on Persian Wikipedia article pages did not reduce administration actions such as blocking and checkuser requests but reduced the number of page protections.
 * Different restriction policies or user preference lead to different trends. Portuguese Wikipedia blocked IP editing on the entire site while Persian Wikipedia only blocked on the article pages. New editor accounts and active logged-in editors improved on Portuguese Wikipedia while they have not changed on Persian Wikipedia.

= References =