Readers/Reader Experience/WE3.2.3 Donate button A/B test

The Donate button A/B test is one of the experiments of the Reader Growth team.

Our hypothesis is that if we run an A/B test experiment on logged out users to display subtle variants of the donation entry point for both mobile and desktop, we will observe a 2% higher number of donations via the treatment paths, as compared to control. Owner is Steph, delegate owner is Jan.

In Q1 25-26, the donate button A/B test on desktop showed promising results to increase clickthrough rate of the “donate” button in the top toolbar. We would like to gradually roll this change out to 100% of English Wikipedia logged out users.

What is this feature?

This feature would entail changing the Donate button entry point to a design that includes a blue heart.

Context

In Q1 2025-26, as part of hypothesis WE 3.2.3 (owner: Steph Toyofuku), we conducted an A/B test on English Wikipedia (desktop) to see whether changing the “Donate” button entry point to a more eye-catching design with a blue heart would encourage more clickthroughs and donations from logged-out users.

At the conclusion of the monthlong A/B test, we saw a significant improvement in the “Donate” button clickthrough rate (~227.47% increase compared to the control), although we did not see an increase in the number of resulting donations. We believe this was due to the very small sample size of users in the A/B test (per experimentation platform limitations) of 0.1%, However, in the past, on a larger scale, we have seen CTR be closely correlated to number of donations.

Thus, we assume that an increase of 200%+ in CTR should lead to an increase in donations if we were to have a large sample size. Due to this, we would like to gradually roll this design out to 100% of logged out users on English Wikipedia on desktop and compare the number of donations received before and after the change in order to validate this assumption.

Experimentation

Phase 1: User testing

Date: FY25-26 Sprint 6 (Q1)

Status: Complete

The experiment targeted only anonymous users with a certain skin version, Minerva skin for mobile web users and Vector-22 for desktop users. In this test, a randomly selected half of anonymous user sessions will be assigned to the “treatment” group, where they will see the new version of donate link. The other randomly selected half will be assigned to the “control” group, where they will see the current version of donate link.

To ensure that the disparity between click through rate and donation rate we saw in the experiment result is not due to user confusion regarding the heart design of the button, Sneha Patel utilized a small-scale user test and verify user intent when interacting with the heart design.
The test went live on Test Wiki 12 August and English Wikipedia 14 August. The experiment ran for a total of four weeks, from 12 August 2025 to 11 September 2025, to assure sufficient data is collected.

Phase 2: Determine rollout parameters

Date: FY25-26 Sprint 9 (Q2)

Status: Ongoing We need to determine a few important details about the rollout and measurement plan:

Metrics to monitor:
- Primary: Clickthrough rate of “Donate” button on Desktop
  - Considerations: we need to check whether we have existing baselines. If not, we can first launch instrumentation and tracking on target Wikis and measure clickthrough baseline for 2 weeks prior to rolling out, for comparison.
- Auxiliary: Absolute clicks and absolute impressions (necessary components of the clickthrough rate calculation)
- Guardrail: Fundraising Analytics to track actual number of donations to ensure there’s no negative impact after rollout
Targeting Wikis beyond English: roll out to all?
- We will explore rolling this design out on other Wikis and go through the localization and design review process to ensure that the heart design works for other languages and cultural contexts (and RTL languages).
Gradual rollout plan
- Align on approach
  - Option 1: Roll out fully on smaller Wikis first, then fully on medium Wikis, then fully on large Wikis
  - Option 2: Roll out incrementally on English to full
Automated testing built in as part of engineering work for ongoing QA.

Phase 3: Develop and deploy on target Wikis and monitor

Date: FY25-26 Sprint 10-11 (Q2)

Status: Planned

Gradually roll out on target Wikis, with roll-back plan as contingency (Jan Drewniak Steph Toyofuku& Hsuanwei Fan). We will set up dashboards to track the clickthrough rates of the new button and monitor for irregularities. Partner with Fundraising Analytics to track donations made through the portal and compare with past YoY and MoM data (if available).

We will explore rolling this design out on other Wikis and go through the localization and design review process to ensure that the heart design works for other languages and cultural contexts (and RTL languages).