Product Analytics/Data products/ptwiki metrics summary Apr2024
The Portuguese Wikipedia has blocked logged-out editing since October 2020, by the local community decision. This means that users who attempt to edit while logged-out are redirected to create an account or log in. This report was commissioned by WMF product leads in December 2023 to revisit the impact of the restriction on the project's health after two prior analyses. The short-term and long-term impacts on each metric are summarized in the table below after comparing the quarter when restriction began to the year before the restriction began, and the same quarter in 2023 to the year before the restriction (2019).
This report contains several observations. The restriction on Portuguese Wikipedia prevented edits that would most likely be reverted and reduced administration blocks and page protections. Some logged-out editors converted to logged-in editors after the restriction, driving new account creation in the short term. Turning off logged-out editing resulted in a significant increase in active editors, which has remained higher over the long term, suggesting that those logged-out editors who did create new accounts were retained as editors over the long term. However, this restriction contributed to a long-term reduction in non-reverted content edits, dampening content creation on Portuguese Wikipedia. Further, the increase in account creation in the short term did not lead to a long-term boost.
In summary, the results of this change were mixed. One of the stated goals of turning off logged-out editing was to reduce moderator burden and reverted edits, and there is evidence that it achieved this goal. On the other hand, there is evidence that this came at the cost of a significant reduction in non-reverted edits, weakening the growth of content in the Portuguese Wikipedia, and potentially leading to other negative long-term effects.
| Category | Metrics | Short-term Impact (By December 2020) | Long-term Impact (By December 2023) |
|---|---|---|---|
| Editors | Number of new accounts | Improved (+19.8%) | The increase has ceased. No change. (-1.7%) |
| Number of active logged-in editors | Improved (+76%) | Improved (+46.1%) | |
| Retention rate | Improved (+19.0%) | No significant change (-7.9%) | |
| Content | Revert rate | Improved (-41.5%) | Improved (-47.1%) |
| Number of net non-reverted content edits | Declined but not significant (-2.6%) | Declined (-22.1%) | |
| Administration | Number of blocks | Improved (-86.3%) | Improved (-92.5%) |
| Number of protected pages | Improved (-77.6%) | Improved (-88.4%) |
Introduction
The Portuguese Wikipedia community implemented a ban on logged-out editing starting from October 4, 2020, following a community vote. This ban was applied across the entire Portuguese Wikipedia, affecting both content pages and talk pages.
In order to assess the impact of this change, the Wikimedia Foundation Product Analytics team conducted analyses of the metrics and published two metrics summary regarding the impact of disabling logged-out editing on the health of Portuguese Wikipedia, based on data up to June 2021[1] and Jun 2022[2].
Two years after our last analysis, Product leads requested a follow-up analysis to evaluate the long-term impact of this decision on the community. As a result, the Product Analytics team analyzed the effects of disabling logged-out editing on Portuguese Wikipedia in 2024.
Methods and considerations
Metrics selection
The analysis delves into metrics that are likely affected by the restriction, primarily examining three areas: editor, content creation, and administration. To measure the impact on editors, we selected three metrics: number of new accounts, number of active user numbers, and new editor retention rate. To assess the impact on content, we concentrated on the revert rate and number of edits. We refined the edit metric by excluding fluctuating factors, such as bot edits, edits reverted within 48 hours, and edits reverting other edits within 48 hours. We measured edits from both the entire Portuguese Wikipedia and specifically from article pages. For evaluating the impact on administration, we reviewed three metrics: frequency of blocks, number of page protections, and number of checkuser checks.
Methods of analysis
In this report, we analyzed the trends in metrics since the restriction was implemented, up until December 2023. To mitigate the influence of seasonal variations, we selected the year before the restriction (FY2019/20, July 2019 - Jun 2020) as the baseline and performed year-over-year comparisons for all subsequent years. The period from July to September 2020 was excluded from the baseline, as some metrics may have already been influenced by discussions surrounding the restriction before it was officially implemented. Since the restriction took effect in Q2 of FY2020/21, we evaluated the short-term impact by comparing FY2020/21 Q2 to FY2019/20 Q2 and assessed the long-term impact by comparing FY2023/24 Q2 to FY2019/20 Q2. Throughout the following report, the term 'study year' refers to FY23/24, during which the data were collected and analyzed.
| Fiscal year | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 (4 years prior) (Baseline) | July - September, 2019 | October - December 2019 | January - March, 2020 | April - June, 2020 |
| FY2020/21 (3 years prior) | July - September, 2020 | October - December 2020 | January - March, 2021 | April - June, 2021 |
| FY2021/22 (2 years prior) | July - September, 2021 | October - December 2021 | January - March, 2022 | April - June, 2022 |
| FY2022/23 (1 year prior) | July - September, 2022 | October - December 2022 | January - March, 2023 | April - June, 2023 |
| FY2023/24 (study year) | July - September, 2023 | October - December 2023 | January - March, 2024 | April - June, 2024 |
We observed that the global pandemic impacted editing patterns across Wikipedias. Therefore, for certain content metrics, we also selected a few reference wikis to comprehend the typical trends during and after the pandemic. The reference wikis were chosen based on two assumptions: 1) that the editing patterns of editors in the same country were similarly affected by the pandemic across different Wikipedia projects, and 2) that editors in neighboring countries were similarly affected by the pandemic.
Based on the edit data spanning from October 2018 to September 2020, 79% of edits on Portuguese Wikipedia (ptwiki) submitted from Brazil, while 15% were from Portugal. Among the top five Wikipedia projects primarily edited in Brazil and Portugal, four reference wikis were chosen: Spanish Wikipedia (eswiki), French Wikipedia (frwiki), Italian Wikipedia (itwiki), German Wikipedia (dewiki). The content contributions to these projects predominantly come from countries in South America and Europe, same as Portuguese Wikipedia. English Wikipedia (enwiki) was excluded due to its status as the largest Wikipedia project, with the majority of edits coming from countries like the USA, United Kingdom, Canada, India, and Australia, which are geographically distant from Brazil and Portugal. Trends on English Wikipedia could be more influenced by global factors affecting these distant countries rather than Brazil and Portugal. Despite being the second-largest Wikipedia project after English Wikipedia, Spanish Wikipedia was still selected as a reference wiki because the majority of its editors are from countries near Portugal and Brazil, such as Spain, Argentina, Chile, Colombia, Peru, Uruguay, Venezuela, and Ecuador.
We also examined other Portuguese-language wiki projects, such as Portuguese Wiktionary (ptwiktionary), Portuguese Wikinews (ptwikinews), Portuguese Wikisource (ptwikisource), and Portuguese Wikiversity (ptwikiversity). However, we did not select any of them as reference candidates due to high variability and extreme fluctuations of measurements from those projects during the comparison period.
We used the trend in edits observed across the four selected wikis (eswiki, frwiki, itwiki, and dewiki) as a reference point to aid in analyzing the results of measuring ptwiki, but not as a definitive benchmark to directly measure ptwiki against. This is for two main reasons: 1) there may be undisclosed factors influencing the reference wikis, and 2) no two wikis are the same. Moreover, historical data has demonstrated that the trend of Portuguese Wikipedia does not correlate well with those of the other Wikipedias prior to the pandemic. The number of edits on Portuguese Wikipedia has been slightly declining since as early as 2018 while the reference wikis did not.
Impacts on editors
Number of new accounts
The number of new accounts is defined as the number of users who register in a given period of time.
In the short term, the number of new accounts surged by 19.8% in Q2 FY20/21 after the restriction was enabled. In the long term, the increase ceased. By Q2 FY2023/24, it returned to the level before the restriction.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 | 12489 | 12487 | 12873 | 16059 |
| FY2020/21 | 12490 | 14960 | 16234 | 16059 |
| FY2021/22 | 14236 | 13230 | 13350 | 13639 |
| FY2022/23 | 12915 | 13536 | 14298 | 14649 |
| FY2023/24 | 12495 | 12269 | - | - |
| 3 year prior compared to 4 years prior | 0.0% | 19.8% | 26.1% | 0.0% |
| 2 year prior compared to 4 years prior | 14.0% | 6.0% | 3.7% | -15.1% |
| 1 year prior compared to 4 years prior | 3.4% | 8.4% | 11.1% | -8.8% |
| Study year compared to 4 years prior | 0.0% | -1.7% | - | - |
Number of active logged-in editors
Number of active user editors is defined as the number of logged-in users who made at least one content edit on Portuguese Wikipedia in the given period of time.
In the short term, number of active user editors increased by 76.0% in Q2 FY2020/21. In the long term, the improvement of active logged-in editors is consistently shown in the following years. But the lift in the study year (FY2023/24) is much smaller than that in 3 years prior (FY2020/21), showing a 46.1% rise compared to the pre-restriction level.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 | 5329 | 5360 | 5270 | 5925 |
| FY2020/21 | 5733 | 9435 | 10407 | 9835 |
| FY2021/22 | 9156 | 8568 | 8388 | 8036 |
| FY2022/23 | 8014 | 8016 | 8849 | 8284 |
| FY2023/24 | 7873 | 7830 | - | - |
| 3 year prior compared to 4 years prior | 7.6% | 76.0% | 97.5% | 66.0% |
| 2 year prior compared to 4 years prior | 71.8% | 59.9% | 59.2% | 35.6% |
| 1 year prior compared to 4 years prior | 50.4% | 49.6% | 67.9% | 39.8% |
| Study year compared to 4 years prior | 47.7% | 46.1% | - | - |
Retention rate
Retention rate is defined as: out of the (non-bot) users who registered in the month and made at least one edit in their first 30 days, the proportion who also edited during their second 30 days.[3] Due to the nature of the definition, this metric surfaces two months after user account creation.
In the short term, the retention rate increased by 19.0% in Q2 FY20/21, driven by the logged-out editor switching to logged-in mode. In the long term, it stayed at the level before the restriction.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 (4 years prior) | 4.5% | 5.6% | 3.7% | 4.6% |
| FY2020/21 (3 years prior) | 5.2% | 6.7% | 5.5% | 5.1% |
| FY2021/22 (2 years prior) | 4.9% | 4.8% | 4.2% | 4.8% |
| FY2022/23 (1 year prior) | 4.1% | 4.8% | 4.7% | 4.4% |
| FY2023/24 (study year) | 4.8% | 5.2% | - | - |
| 3 years prior compared to 4 years prior (YoY) | 13.9% | 19.0% | 47.1% | 11.3% |
| 2 years prior compared to 4 years prior (YoY) | 8.1% | -15.0% | 14.1% | 4.2% |
| 1 year prior compared to 4 years prior (YoY) | -9.4% | -14.4% | 26.5% | -3.6% |
| Study year compared to 4 years prior (YoY) | 6.4% | -7.9% | - | - |
| 3 years prior compared to 4 years prior (percentage diff) | 0.6% | 1.1% | 1.7% | 0.5% |
| 2 years prior compared to 4 years prior (percentage diff) | 0.4% | -0.8% | 0.5% | 0.2% |
| 1 year prior compared to 4 years prior (percentage diff) | -0.4% | -0.8% | 1.0% | -0.2% |
| Study year compared to 4 years prior (percentage diff) | 0.3% | -0.4% | - | - |
Takeaways
In the short term, disabling logged-out editing led to a brief surge in account creation. In addition, the retention rate among newcomers who joined immediately after the restriction was slightly higher. However, this effect did not persist in the long term.
Moreover, turning off logged-out editing resulted in a significant increase in active editors, with a 76% rise in the short term. Although this boost diminished over the four years following the restriction, it remained persistently higher than before the restriction. This indicates that some anonymous editors were inclined to register new accounts and continue actively editing on Portuguese Wikipedia.
Impacts on content
Revert rate
Revert rate is defined as the proportion of edits reverted within 48 hours of being published out of total edits in the given period of time. Low revert rate is considered an improvement.
In the short term, the revert rate decreased by 41.5% in Q2 FY2020/21. In the long term, the improvement in revert rate is consistently shown in FY2021/22, FY2022/23, and FY 2023/24.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 (4 years prior) | 9.75% | 10.27% | 8.39% | 10.64% |
| FY2020/21 (3 years prior) | 9.17% | 6.01% | 6.01% | 5.67% |
| FY2021/22 (2 years prior) | 5.72% | 5.73% | 5.46% | 5.08% |
| FY2022/23 (1 year prior) | 5.25% | 5.76% | 5.05% | 5.38% |
| FY2023/24 (study year) | 5.72% | 5.43% | - | - |
| 3 years prior compared to 4 years prior (YoY) | -5.9% | -41.5% | -28.4% | -46.7% |
| 2 years prior compared to 4 years prior (YoY) | -41.3% | -44.2% | -34.9% | -52.3% |
| 1 year prior compared to 4 years prior (YoY) | -46.2% | -43.9% | -39.8% | -49.4% |
| Study year compared to 4 years prior (YoY) | -41.3% | -47.1% | - | - |
| 3 years prior compared to 4 years prior (percentage diff) | -0.58% | -4.26% | -2.38% | -4.97% |
| 2 years prior compared to 4 years prior (percentage diff) | -4.03% | -4.54% | -2.93% | -5.56% |
| 1 year prior compared to 4 years prior (percentage diff) | -4.50% | -4.51% | -3.34% | -5.26% |
| Study year compared to 4 years prior (percentage diff) | -4.03% | -4.84% | - | - |
Net non-reverted edits
Net non-reverted edits are defined as the number of edits that were not reverted within 48 hours, excluding bot edits, edits reverted in 48 hours, and edits that reverted other edits in 48 hours. They are considered 'good' edits by users.
In the short term, the net non-reverted edits decreased by 0.9% in Q2 FY2020/21. However, in the long term, the decline is more pronounced. The number of net non-reverted edits is 20.2% lower in the study year (FY2023/24) than four years prior.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2018/19 (5 years prior) | 178590 | 165561 | 182899 | 194156 |
| FY2019/20 (4 years prior) | 201066 | 180069 | 179614 | 185231 |
| FY2020/21 (3 years prior) | 221766 | 178424 | 183291 | 196546 |
| FY2021/22 (2 years prior) | 178895 | 165470 | 164741 | 166639 |
| FY2022/23 (1 year prior) | 156987 | 145797 | 167994 | 147606 |
| FY2023/24 (study year) | 144956 | 143661 | - | - |
| 3 years prior compared to 4 years prior | 10.3% | -0.9% | 2.0% | 6.1% |
| 2 years prior compared to 4 years prior | -11.0% | -8.1% | -8.3% | -10.0% |
| 1 year prior compared to 4 years prior | -21.9% | -19.0% | -6.5% | -20.3% |
| Study year compared to 4 years prior | -27.9% | -20.2% | - | - |
Net non-reverted content edits
Net non-reverted content edits are defined as the number of content (main-namespace) edits that were not reverted within 48 hours, excluding bot edits, reverted edits, and edits that reverted other edits. Content edits are identified based on whether they are from a content namespace.
In the short term, the net non-reverted content edits decreased by 2.6% in Q2 FY2020/21. However, the decline is more significant in the long term. The number of net non-reverted content edits has decreased by 22.1% in the study year (FY2023/24) compared to four years prior (FY2019/20).
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2018/19 (5 years prior) | 133913 | 125289 | 134459 | 147502 |
| FY2019/20 (4 years prior) | 154321 | 132012 | 133236 | 131285 |
| FY2020/21 (3 years prior) | 166618 | 128595 | 123720 | 132507 |
| FY2021/22 (2 years prior) | 120482 | 109471 | 111890 | 113829 |
| FY2022/23 (1 year prior) | 105311 | 98015 | 113500 | 98271 |
| FY2023/24 (study year) | 98442 | 102844 | - | - |
| 3 years prior compared to 4 years prior | 8.0% | -2.6% | -7.1% | 0.9% |
| 2 years prior compared to 4 years prior | -21.9% | -17.1% | -16.0% | -13.3% |
| 1 year prior compared to 4 years prior | -31.8% | -25.8% | -14.8% | -25.1% |
| Study year compared to 4 years prior | -36.2% | -22.1% | - | - |
Compared with the other wikis, in Q2 FY2020/21, four reference wikis show an average increase of 14.3%, while growth is not present on Portuguese Wikipedia. In Q2 FY2023/24, edits on four reference wikis has returned to pre-pandemic levels, though still 2.1% higher on average than for year prior (Q2 FY2019/20). Portuguese Wikipedia is significantly lower than pre-pandemic levels by 22.1%. This suggests that the editing boost from pandemic behavior shifts might have offset the decline caused by turning off logged-out editing, making the decline less obvious in Q2 FY2020/21. However, the decline starts to surface in the post-pandemic period.
- Net Non-Reverted Content Edits on Four Reference Wikis
-
Spanish Wikipedia -- Increased by 9.1% compared to the pre-restriction level. -
German Wikipedia -- Increased by 4.3% compared to the pre-restriction level. -
French Wikipedia -- Returned to the pre-pandemic level. -
Italian Wikipedia -- Returned to the pre-pandemic level.
Takeaways
Disabling logged-out editing resulted in a significant decrease in the revert rate in both the short term and long term. As reverting is commonly used to fight vandalism, this suggests that by disabling logged-out editing, vandalism was suppressed to some extent.
Due to the intervention of the global pandemic, disabling logged-out editing did not exhibit a significant impact on edits in the short term. The increase in editing spurred by pandemic behavior shifts offset the decline caused by turning off logged-out editing, thus making the decrease less apparent in Q2 FY2020/21. Nonetheless, as we have transitioned into the post-pandemic period, editing trends have reverted to their original levels, revealing a negative impact on content edits. The number of net non-reverted content edits in Q2 of the study year (FY2023/24) is 22.1% lower than in the same quarter four years prior (FY2019/20).
Impacts on administration
Number of blocks
Number of blocks is defined as the number of block and reblock actions on accounts or IPs or IP ranges in a given period of time. Fewer blocks is considered an improvement.
In the short term, the number of blocks decreased by 86.3% in Q2 FY2020/21 after the restriction was enabled. The improvement in the number of blocks is consistently shown in the long term. In Q2 FY2023/24 it is 92.5% lower than the level before the restriction.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 (4 years prior) | 5952 | 9278 | 4361 | 4207 |
| FY2020/21 (3 years prior) | 3596 | 1274 | 931 | 991 |
| FY2021/22 (2 years prior) | 921 | 833 | 930 | 841 |
| FY2022/23 (1 year prior) | 878 | 891 | 1021 | 1126 |
| FY2023/24 (study year) | 833 | 692 | - | - |
| 3 years prior compared to 4 years prior | -39.6% | -86.3% | -78.7% | -76.4% |
| 2 years prior compared to 4 years prior | -84.5% | -91.0% | -78.7% | -80.0% |
| 1 year prior compared to 4 years prior | -85.2% | -90.4% | -76.6% | -73.2% |
| Study year compared to 4 years prior | -86.0% | -92.5% | - | - |
Number of protected pages
Number of protected pages is defined as the number of unique pages protected in a given period of time. Having fewer protected pages is considered an improvement.
In the short term, the number of protected pages decreased by 77.6% in Q2 FY 2020/21, after the restriction was enabled. The improvement in the number of protected pages is consistently shown in the long term. In Q2 2023/24, it’s 88.4% lower than in FY 2019/20.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 (4 years prior) | 375 | 825 | 468 | 384 |
| FY2020/21 (3 years prior) | 1045 | 185 | 148 | 232 |
| FY2021/22 (2 years prior) | 134 | 112 | 132 | 137 |
| FY2022/23 (1 year prior) | 126 | 112 | 173 | 111 |
| FY2023/24 (study year) | 116 | 96 | - | - |
| 3 years prior compared to 4 years prior | 178.7% | -77.6% | -68.4% | -39.6% |
| 2 years prior compared to 4 years prior | -64.3% | -86.4% | -71.8% | -64.3% |
| 1 year prior compared to 4 years prior | -66.4% | -86.4% | -63.0% | -71.1% |
| Study year compared to 4 years prior | -69.1% | -88.4% | - | - |
Number of CheckUser checks
Number of CheckUser checks is defined as the number of CheckUser checks made by administrators in a given period of time. The metric is for curiosity observation. The historical trend shows high volatility. The number of checkuser checks is 27.2% fewer in FY 2023/24 than FY2019/20.
| Monthly Average | Q1 | Q2 | Q3 | Q4 |
|---|---|---|---|---|
| FY2019/20 (4 years prior) | 179 | 227 | 288 | 372 |
| FY2020/21 (3 years prior) | 314 | 381 | 259 | 204 |
| FY2021/22 (2 years prior) | 131 | 188 | 124 | 119 |
| FY2022/23 (1 year prior) | 236 | 146 | 144 | 246 |
| FY2023/24 (study year) | 299 | 165 | - | - |
| 3 years prior compared to 4 years prior | 75.4% | 67.8% | -10.1% | -45.2% |
| 2 years prior compared to 4 years prior | -26.8% | -17.2% | -56.9% | -68.0% |
| 1 year prior compared to 4 years prior | 31.8% | -35.7% | -50.0% | -33.9% |
| Study year compared to 4 years prior | 67.0% | -27.3% | - | - |
Takeaways
The majority of administrative actions are implemented to address vandalism and anti-harassment issues. We observed a significant decrease in the number of blocks and number of page protections in both short term and long term. Disabling logged-out editing led to a reduction in the administrative workload.
References
- ↑ https://meta.wikimedia.org/wiki/IP_Editing:_Privacy_Enhancement_and_Abuse_Mitigation/IP_Editing_Restriction_Study/Portuguese_Wikipedia
- ↑ https://www.mediawiki.org/wiki/Product_Analytics/Data_products/ptwiki_metrics_summary_Jun2022
- ↑ https://meta.wikimedia.org/wiki/Research_and_Decision_Science/Data_glossary