Wikimedia Apps/Team/Android/Private Donor Recognition Experiment

As part of our contributions to 2024/2025 Annual Plan, the Android team will explore various ways to increase the number of donations via touchpoints outside the annual banner and email appeals per platform. This project page documents the Android team’s experimentation related to the Wikimedia Foundation 2024-2025 Annual Plan, specifically the Wiki Experiences 3.2 Key Result.

Summary

Based on feedback and data, our focus is on increasing the number of donations beyond the methods the Foundation has relied upon in the past, specifically the annual banner appeals. We want to show that by investing in more integrated donor experiences, we can sustain our work and expand our impact by providing an alternative to donors and potential donors that are unresponsive to banner appeals.

Our existing donors would like to be recognized for their contributions and receive a tailored approach to engagement. Our current experiences, beyond email, fail to differentiate from existing recurring donors, one time donors, and potential donors.

Additionally we do not associate donor status with user names. We have an opportunity to learn if recognizing current and potential donors for their contributions, much like what we offer editors, would offer value to our users and if it opens the ability for more intentional campaigns thus it is worth investing in a cross platform system.

Background

Wiki Experiences 3: Consumer experience (Reading & Media)

Under the Wikimedia Foundation's Infrastructure Goal, and within the group of objectives focused on Wiki Experiences, is an objective related to improving the experience of consumers:

Wiki Experiences 3: Consumer experience Objective: A new generation of consumers arrives at Wikipedia to discover a preferred destination for discovering, engaging, and building a lasting connection with encyclopedic content.

Wiki Experiences 3.2 (WE3.2) Key Result: 50% increase in the number of donations via touchpoints outside the annual banner and email appeals per platform.

Several Wikimedia Foundation teams are committed to working on projects under the WE3.2 Key Result: Draft Hypotheses.

Android team hypothesis	Timeline	Phabricator epic
Wiki Experiences 3.2.3: If we update the contributions page in the app to include an active badge for someone that is an app donor and display an inactive state with a prompt to donate for someone that decided not to donate in app, we will learn if this recognition is of value to current donors and encourages behavior of donating for prospective donors, informing if it is worth expanding on the concept of private donor badges or abandoning it.	1 October 2024 - 31 December 2024	T376303

Wikimedia Foundation teams are approaching annual planning more iteratively this year, so rather than committing to larger year-long projects, this hypothesis is fairly narrow in scope. This should allow us to learn from a brief experiment and deliver value in smaller increments throughout the year, while also ensuring we have the flexibility to pivot as we learn.

Community

We’re using in-app surveys to collect feedback from app users.

• Donors in France, and Netherlands.

Designs

• 
  Initial screen (Donor)
• 
  Initial screen (Not a donor/unknown)
• 
  Overflow menu
• 
  Logged-out dashboard
• 
  Donor history unknown
• 
  Donor history (donor=yes)
• 
  Logged-in dashboard
• 
  Survey ask

Measurement and Results

How will we know we were successful?

The team is planning to run an experiment for 20 days to evaluate our hypothesis, along with running a survey to understand user satisfaction and feedback. Our experiment groups are Android app donors in the Netherlands and France. We will validate our hypothesis for the experiment groups with the key indicators and guardrails listed below:

Validation

Current and Prospective Donors

• KR 1: 30% Badge and Donor Additions interaction rate
• 1.1 30% CTR to contributions Dashboard page
• 1.2 2% Save changes
• KR 2: 55% of survey respondents reflect desire for permanence (Qualitative)
• KR 3: 50 donations as a result of feature
• Current Donors
• KR 4: 30% of users that click Contributions Dashboard make an update

Prospective Donors

• KR 5: 40 donations from non donors or unknown donors to donors as a result of feature

Guardrails:

• GR 1: 65% of negative sentiments by in-app users of feature
• GR 2: 10 negative comments on talk page by community members from target audience

Curiosities:

• CR 1: Does showing the edit tab to logged out users result in more account activation?
• CR 2: How many times did we guess someone was a donor correctly vs users had to update their status?
• CR 3: Do users convey frustration with sharing their information as opposed to us already having access? (Qualitative)
• CR 4: Is there alternative placement we should consider than the contributions dashboard? (Qualitative)
• CR5: What proportion of users that made updates were Non Donors vs Donors?
• CR6: How many users updated their icon via settings?

After conducting data analysis we will share the results of the experiment on this page, and in partnership with the community determine if there should be further investment and rerelease of content suggestions or if we should scale another hypothesis instead.

Technical Constraints

• Unable to get cross platform donations
• Unable to know insights if users donate via web
• The app cannot know whether the user has made donations in the past, or what amounts they were.
• If the user signs up for a recurring donation through the app, and then later cancels it (which is done outside of the app), the app won't be able to know about it.
• If the user uninstalls and reinstalls the app, or clears the app's data, their local contribution history will be lost.
• If the user logs in on another device, their contribution history will be empty.

Updates

• Monthly updates can be found on our main updates page.
• A 20-day results from this experiment that ran November 21 - December 11:
  • The experiment was ran in France and the Netherlands, in English, French, and Dutch.
  • For Current and Prospective Donors
    • KR 1.1: 30% click through to the contributions Dashboard page
      • Actual: 3.5% of users who saw the prompt to visit the contributions dashboard clicked through to it. The donor-prompt shown to users who have just donated or chosen “I already donated” on a banner received a higher click-through-rate than the general prompt.
    • KR 1.2: 2% Save changes
      • Actual: 0.6% of users who saw the invitation to the dashboard actually saved changes to their donor status after seeing the prompt for the dashboard. When we look at just users who actually clicked through to the dashboard, 37.4% of users saved changes to their Donor Stats.
    • KR 2: 55% of survey respondents reflect desire for permanence
      • Actual: 81% of survey respondents wanted the donor dashboard to exist on either one or multiple platforms. 70% of respondents wanted it to be available across platforms. 19% did not want it to be a permanent feature.
    • KR 3: 50 donations as a result of feature
      • Actual: We saw fewer than 10 donations made through the feature.
  • Current Donors
    • KR 4: 30% of users that click Contributions Dashboard make an update
      • Actual: Our hypothesis was that this will be higher than KR 1.2, because people will be more likely to self-identify as a donor than a non-donors. This was confirmed, 7.3% of Current/Known Donors made Donor History Updates, compared to overall where only 0.6% of all uniques saved changes.
  • Prospective Donors
    • KR 5: 40 donations from non donors or unknown donors become donors as a result of feature
      • Actual: We did not meet this goal, 50% of donations came from new or unknown donors.
  • Guardrails
    • GR 1: 65% of negative sentiments by in-app users of feature
      • Actual: We met this guardrail: only 5% of survey respondents expressed that they did not like the concept of being recognized as a donor. 28% of survey respondents expressed that they did not like the concept of unlocking extra features as a donor.
  • Curiosities
    • CR 1: Does showing the edit tab to logged out users result in more account activation?
      • This feature showed information within the Contribution tab to logged-out users, where previously it was only an invitation to log-in. We saw 36 users create accounts after seeing Contribution Tab, 22% of all new accounts during this period in this audience.
    • CR 3: Do users convey frustration with sharing their information as opposed to us already having access?
      • The contributions dashboard did not know about donations made outside the apps, so we asked users to make updates manually.
      • 87.7% of users wanted to be able to view their donor stats, and 9% did not want to see any data related to their donation patterns.
      • 45.6% of users did not want to have to edit their stats manually, suggesting a desire for it to happen automatically.
    • CR6: How many users updated their icon via settings?
      • For those with a donor status, we offered them the chance to update to a (temporary) custom app icon. 39% of users updated their icon.
• Major lessons learned from this feature.
  • Donors who engaged with the feature reacted positively to both the concept of donor recognition and the tangible recognition provided through this experiment with both the dashboard and a dedicated app icon for donors.
  • It was interesting that there was more muted enthusiasm for donor exclusive features.
  • Care will need to be taken in the future in striking the right balance in the framing between recognition and exclusivity.
  • There was a clearly expressed desire by those who engaged with the feature that this concept should be available across all platforms, and that it shouldn't require manual intervention by the user. It should be automatic.
  • Ideally we would have a privacy respecting mechanism to be able to identify donors across the platforms in order to bring additional value to users in a more consistent and efficient manner. This would increase the overall effectiveness of donor recognition efforts.