Extension:Popups/Fundraising test 2
Following on from the first test in Hungary, in October/November 2016 the Fundraising and Reading teams ran a second experiment to see if enabling Hovercards had an effect on banner fundraising.
Methodology
editWe selected the Italian Wikipedia for our next experiment, since it was a larger Wikipedia where the community was receptive to a test of Hovercards, and we had already planned fundraising in Italy for October/November.
Hovercards were enabled on the Italian Wikipedia for 20% of anonymous users on September 23. On October 11 the fundraising team began running their planned banners on the Italian Wikipedia desktop site in Italy, with the usual campaign setup:
- anonymous users only
- users see one large banner (example) on their first visit to the site, and then smaller banners (example) on subsequent visits
- two weeks after launch (October 25) the number of banners a user will see is limited to a maximum of 3
- four weeks after launch (November 8) the campaign ends and banners are disabled
We used our standard fundraising banners for this test, translated into Italian and with some minor technical modifications to log whether Hovercards were enabled or disabled for the user.
Results
editFull data can be seen here.
Large banners
edit| donations | impressions | dons/i | clicks | clicks/i | dons/clicks | avg | |
|---|---|---|---|---|---|---|---|
| large banners hovercards disabled | 49,649 | 16,007,800 | 0.00310155 | 81,462 | 0.0050889 | 60.95% | $7.30 |
| large banners hovercards enabled | 12,497 | 3,995,621 | 0.00312767 | 20,333 | 0.0050888 | 61.46% | $7.31 |
| difference[1] | +0.7% | -0.2% | +0.8% | -0.2% | 0.0% | +0.8% | +0.1% |
Performance comparison - Hovercards gives between 1.1% decrease and 2.8% increase in number of donations / impressions (with 95% confidence) i.e. likely no change.
Small banners
edit| donations | impressions | dons/i | clicks | clicks/i | dons/clicks | avg | |
|---|---|---|---|---|---|---|---|
| small banners hovercards disabled | 31,178 | 38,491,507 | 0.00081000 | 55,960 | 0.0014538 | 55.71% | $7.00 |
| small banners hovercards enabled | 7,723 | 9,283,869 | 0.00083187 | 13,825 | 0.0014891 | 55.86% | $7.11 |
| difference[1] | -0.9% | -3.7% | 2.7% | -1.2% | 2.4% | 0.3% | 1.6% |
Performance comparison - Hovercards gives between 0.18% and 5.3% increase in number of donations / impressions (with 95% confidence)
Total
edit| donations | impressions | dons/i | clicks | clicks/i | dons/clicks | avg | |
|---|---|---|---|---|---|---|---|
| total hovercards disabled | 80,827 | 54,499,307 | 0.001483 | 137,422 | 0.002522 | 58.82% | $7.18 |
| total hovercards enabled | 20,220 | 13,279,490 | 0.001523 | 34,158 | 0.002572 | 59.20% | $7.23 |
| difference[1] | 0.1% | -2.6% | 2.7% | -0.6% | 2.0% | 0.6% | 0.7% |
Performance comparison - Hovercards gives between 1.1% and 4.2% increase in number of donations / impressions (with 95% confidence)
Conclusions
editAs expected, and as was seen in the Hungary test, enabling Hovercards in Italy slightly decreased the number of small fundraising banners seen (by approximately 4%). However again there was also a small but statistically significant increase in donation rate per banner, perhaps because users remain on the same page seeing the banner for longer. The overall effect is that enabling Hovercards appears to have a negligible effect on total donations.