Extension:Popups/Fundraising test 1
In June/July 2016, the Fundraising and Reading teams ran an experiment to see if enabling Hovercards had an effect on banner fundraising.
Methodology
editWe selected the Hungarian Wikipedia for the first experiment, since there was already fundraising planned in Hungary in June/July, and it is a mid-sized Wikipedia where the community were receptive to a test of Hovercards.
Hovercards were enabled on the Hungarian Wikipedia for 50% of users on June 7. On June 15 the fundraising team began running banners on the Hungarian Wikipedia desktop site in Hungary, with the usual campaign setup:
- anonymous users only
- users see one large banner (example) on their first visit to the site, and then smaller banners (example) on subsequent visits
- two weeks after launch (29 June) the number of banners a user will see is limited to a maximum of 3
- four weeks after launch (13 July) the campaign ends and banners are disabled
We used our standard fundraising banners for this test, translated into Hungarian and with some minor technical modifications to log whether Hovercards were enabled or disabled for the user.
Results
editFull data can be found here
Large banners
edit| donations | impressions | dons/i | clicks | clicks/i | dons/clicks | avg | |
|---|---|---|---|---|---|---|---|
| large banners hovercards disabled | 2,715 | 1,330,583 | 0.002040 | 7,515 | 0.005648 | 36.13% | $4.65 |
| large banners hovercards enabled | 2,853 | 1,309,633 | 0.002178 | 7,613 | 0.005813 | 37.48% | $4.53 |
| +5.08% | -1.57% | +1.30% |
Performance comparison - Hovercards gives between 1.3% and 12% improvement in number of donations / impressions (with 95% confidence)
Small banners
edit| donations | impressions | dons/i | clicks | clicks/i | dons/clicks | avg | |
|---|---|---|---|---|---|---|---|
| small banners hovercards disabled | 1,351 | 2,566,983 | 0.000526 | 3,642 | 0.001419 | 37.10% | $4.76 |
| small banners hovercards enabled | 1,316 | 2,265,851 | 0.000581 | 3,543 | 0.001564 | 37.14% | $4.94 |
| -2.59% | -11.73% | -2.72% |
Performance difference - Hovercards gives between 2.4% and 18% improvement in number of donations / impressions (with 95% confidence)
Total
edit| donations | impressions | dons/i | clicks | clicks/i | dons/clicks | avg | |
|---|---|---|---|---|---|---|---|
| total hovercards disabled | 4,066 | 3,897,566 | 0.001043 | 11,157 | 0.002863 | 36.44% | $4.68 |
| total hovercards enabled | 4,169 | 3,575,484 | 0.001166 | 11,156 | 0.003120 | 37.37% | $4.66 |
| +2.53% | -8.26% | -0.01% |
Performance difference - Hovercards gives between 7.2% and 16% improvement in number of donations / impressions (with 95% confidence)
Conclusions
editAs expected, the Hovercards feature reduces the number of pages viewed by a user, and hence the number of the later banners seen. This results in showing approximately 8% fewer banners over the course of the campaign. However it also results in an increased donation rate per banner, perhaps because users remain on the same page seeing the banner for longer. Therefore it appears that this feature did not have a negative effect on the fundraising campaign. While these results are promising, we know that behaviour can vary across languages and countries, and would like to get further data.
Next steps
editWe are planning to run a similar test on a larger Wikipedia, possibly Italian in October.