Extension:Popups/Fundraising test 1

In June/July 2016, the Fundraising and Reading teams ran an experiment to see if enabling Hovercards had an effect on banner fundraising.

Methodology

We selected the Hungarian Wikipedia for the first experiment, since there was already fundraising planned in Hungary in June/July, and it is a mid-sized Wikipedia where the community were receptive to a test of Hovercards.

Hovercards were enabled on the Hungarian Wikipedia for 50% of users on June 7. On June 15 the fundraising team began running banners on the Hungarian Wikipedia desktop site in Hungary, with the usual campaign setup:

• anonymous users only
• users see one large banner (example) on their first visit to the site, and then smaller banners (example) on subsequent visits
• two weeks after launch (29 June) the number of banners a user will see is limited to a maximum of 3
• four weeks after launch (13 July) the campaign ends and banners are disabled

We used our standard fundraising banners for this test, translated into Hungarian and with some minor technical modifications to log whether Hovercards were enabled or disabled for the user.

Results

Full data can be found here

Large banners

	donations	impressions	dons/i	clicks	clicks/i	dons/clicks	avg
large banners hovercards disabled	2,715	1,330,583	0.002040	7,515	0.005648	36.13%	$4.65
large banners hovercards enabled	2,853	1,309,633	0.002178	7,613	0.005813	37.48%	$4.53
	+5.08%	-1.57%		+1.30%

Performance comparison - Hovercards gives between 1.3% and 12% improvement in number of donations / impressions (with 95% confidence)

Small banners

	donations	impressions	dons/i	clicks	clicks/i	dons/clicks	avg
small banners hovercards disabled	1,351	2,566,983	0.000526	3,642	0.001419	37.10%	$4.76
small banners hovercards enabled	1,316	2,265,851	0.000581	3,543	0.001564	37.14%	$4.94
	-2.59%	-11.73%		-2.72%

Performance difference - Hovercards gives between 2.4% and 18% improvement in number of donations / impressions (with 95% confidence)

Total

	donations	impressions	dons/i	clicks	clicks/i	dons/clicks	avg
total hovercards disabled	4,066	3,897,566	0.001043	11,157	0.002863	36.44%	$4.68
total hovercards enabled	4,169	3,575,484	0.001166	11,156	0.003120	37.37%	$4.66
	+2.53%	-8.26%		-0.01%

Performance difference - Hovercards gives between 7.2% and 16% improvement in number of donations / impressions (with 95% confidence)

Conclusions

As expected, the Hovercards feature reduces the number of pages viewed by a user, and hence the number of the later banners seen. This results in showing approximately 8% fewer banners over the course of the campaign. However it also results in an increased donation rate per banner, perhaps because users remain on the same page seeing the banner for longer. Therefore it appears that this feature did not have a negative effect on the fundraising campaign. While these results are promising, we know that behaviour can vary across languages and countries, and would like to get further data.

Next steps

We are planning to run a similar test on a larger Wikipedia, possibly Italian in October.