Talk:Growth/Newcomer experience projects

Encouraging donors to edit (Thank you page experiment)

Donor Thank you page with a "Try editing Wikipedia" prompt

The Wikimedia Foundation's Fundraising team started a community discussion so volunteers can learn more about fundraising and share ideas for how we can improve the 2023 English fundraising campaign: Wikipedia:Fundraising/2023 banners

One theme that emerged from volunteer feedback was around the need to recruit more editors, rather than just fundraise: "We need editors, not money."

The Fundraising team reached out the Growth team to ask if we would be able to expand the Thank you page experiment to English Wikipedia if the community is interested. The Growth team has committed to the work, and providing edit funnel analysis afterwards if there is support for trying this on English Wikipedia.

Previously we tested a revised Thank you page with a “Try editing Wikipedia” call to action with donors in Latin America, India, and South Africa. And then in early 2023, we scaled the Thank you page experiment to Swedish, Italian, Japanese, French, and Dutch Wikipedias.

Account creation page for donors

Donors who created an account were sent to a custom account creation page for donors, and then received the standard Growth features and onboarding.

Here’s what we learned:

• Approximately 7% of donors in these markets showed interest in editing immediately after donating, based on the estimated click-through rate from the Thank you page.
• The landing page achieved a 45.1% account creation rate, which is a promising result compared to other channels.
• Only 4.6% of the accounts created right after donating started editing within 24 hours of their creation, which is significantly lower than organic registrations.

I'm starting this discussion to see if there is support for expanding this revised Thank you page and donor onboarding at English Wikipedia. What questions do you have? What concerns do you have? Do you have any suggestions for improving this experiment? - KStoller-WMF (talk) 22:11, 3 August 2023 (UTC)

Hello! I left a few thoughts on en:wp. My main feedback is threefold:

1. A "Thank you!" campaign (or any messaging campaign) that was A/B tested for effective recruiting would be fantastic.
2. A/B testing of the newbie-first-edit funnel would be a good companion process; I'd strongly like to see people invited to make edits, reviews, or other engagement with content in the first minute after choosing to start contributing. (Think of a smooth Magnus tool or a vote for great photos; something that can be reduced to a quick evaluation and annotation, while also showcasing the breadth and depth of the projects)
3. Topical messages that filter for readers interested in a particular hobby or topic area might have dramatically different effects, and over time would help us build up an [opt-in and] well-categorized network of supporters and participants by area of enthusiasm, which is closer to what permanent community sustainability looks like
4. When is the last time we tried this for a banner campaign, other than "Wiki Loves X"? Which of those have targeted logged-out readers? What sort of A/B testing have we done on them?  :)

Thanks for the work on the past experiments, and for starting this thread. Sj (talk) 23:34, 3 August 2023 (UTC)

Thanks for the feedback, User:Sj!

The Newcomer Homepage will suggest edits that relate to a newcomer interests

1. I chatted more about A/B testing options with my engineering team, and although we can certainly A/B test certain aspects of a Thank you page experiment, we likely can't conduct the A/B test that I would find most insightful due to how we handle user data and limit tracking. For example, I would like to show the new page (with the “Try editing Wikipedia” call to action) to 50% of donors and compare that to the 50% of donors that don't see that call to action. However, we simply don't track logged out traffic in that way. Once a donor navigates away from the Thank you page, if they aren't navigating to create an account (via that one specific link) we aren't tracking that. Is that the A/B test you were envisioning, or is there another experiment you think we should consider?
2. Yes! This sounds similar to Growth's structured tasks. These tasks break down the editing process to a very easy, guided process for brand new account holders. They are also suggested on the Newcomer homepage, which new accounts are guided to after account creation. These tasks help more new account holders edit for the first time. We A/B tested this feature: Add a link Experiment analysis. However, these tasks aren't released on English Wikipedia yet. We hope to release Add a link to English Wikipedia by the end of the year (communities can then disable or adjust the task via Special:EditGrowthConfig if needed). These tasks make initial editing very easy, but that of course leads to more patrolling for experienced editors. Do you think this task will work for English Wikipedia?
3. Will you tell me more about this idea? I've considered if we could eventually use newcomer interests to help surface related Wiki Projects or edit-a-thons in some way, but curious if you have any specific ideas around this.
4. Last year was the last time we tested a banner to recruit editors: In a 2022 experiment, we tested a banner that encouraged readers to edit as part of the Thank you page experiment. We concluded that this wasn't an effective means of recruiting editors. You can see the full results in the second half of this report: “Create an account” Invitations on Fundraising Thank You Pages and Thank You Banners. Despite over 50 million banner impressions, the banner only led to 492 "constructive activations" (a constructive activation is defined a new account editing within 24 hours of registration and that edit not being reverted within 48 hours). We A/B tested a few changes to the banner language to improve click-through rate, with limited impact. Ultimately we concluded the banners didn't seem like an effective way to encourage Readers to try editing and that the Growth team should utilize our time testing other methods for new editor recruitment and continue our efforts to ensure we engage and retain new editors. The Thank you page call to action lead to far more constructive activations (1,339), despite being displayed to a much smaller group of people.

Can you think of other English Wikipedia editors who might want to participate in this discussion and decision about whether we test a “Try editing Wikipedia” call to action on this year's Donor Thank you page? - KStoller-WMF (talk) 21:51, 7 August 2023 (UTC)

Thanks for these updates @KStoller-WMF: sorry I didn't see the above earlier ;) And thanks for the excellent data from this year's thank you campaign. Various responses + thought experiments below (some repeated from other places, some likely already considered, &c Sj (talk))

1. A/B testing thoughts:
   • Please revise policies to allow short-timeframe session tracking. Escalate to a strategic decision if necessary. This is needed for some categories of user flow analysis, and can be done in a fairly privacy-preserving way. This seems like an absolutely critical piece of site tooling that deserves modifying our risk and data policies to enable.
   • Choose a suitable small cohort-size, well under 1% of donors or 0.1% of visitors, to show a wide range of tests to.
   • Choose a slimmed-down presentation of a single call to action to optimize the language for each call. For things like editing or visiting the shop, you could try this on banners for non-donor readers, including outside the fundraising season, to get a baseline.
   • Try a thank-you page that has only one call to action for each call (the example page I saw had 5-6, some requiring scrolling down on desktop and farther on mobile). Try having call to action on the first screen on mobile, in a convenient place to click. (over the background image?) Try having each as the first call on a page with all five.
   • Share data on how many people followed each call to action.
   • Try a version that opens the calls to action in new tabs
   • Try a version with images + simple text/alt-text rather than a sentence of text for each call.
2. Giving people an immediate way to edit after clicking a "Try editing" button:
   • No need to take people to account-creation first. Take them directly to a simple edit, then another.
   • For the purposes of editing, assign a temp account. After a few edits, invite username/password selection with a popup, without breaking attention to contributing. [the session length and engagement replaces the captcha, no need for email though you can offer OAuth signin via an email provider]
   • The 'add a link' experiment is a great idea -- but it's still a dozen clicks before you can edit! Get people editing in one click; the best tutorials are by doing. The demo I saw offers too many options up front, and also has an obligatory popup every time you decline to add a link, while only letting you consider one matching article at a time (the meat of the edit) rather than choosing from a dropdown as the VE normally does. So: both more red tape and less depth of contribution.
   • Early in the intake flow, new editors should be invited to review the edits of other new editors; self-moderation is essential to sustaining new contribs.
   • Somehow, the 'add a link' experiment from 2021 still hasn't made it to larger wikis. Speeding this sort of improvement should be higher priority.
3. Topical reader/donor campaigns
   • Reader banner: "Support our coverage of breaking news / science / food / football!" with suitable image. The subject of the banner should be tuned to the category of article that it is shown on. Taking readers to a donation page with language focused on that subject area, editing campaigns in that area, &c. A major wikiproject in the topic could help with the language, task list, and ongoing engagement with those readers. And you could classify donors by the call that drew them in, perhaps following up w/ a short poll on the topic to see how relevant the topic was to their interest (and experience)
   • Donor thank you: A variant of "Try editing!" flow that dynamically updates offered topics as editors choose what they are working on. Each task is just a single paragraph, letting them complete 5-10 quick tasks while getting a loose sense of topical interest, then zooming out a bit and inviting them to join a project to meet others working on the same topic. (opt-in by projects that want to engage with a stream of newbies) Implicit pingflow here would be choosing a username (+ optional email) at this point so you could get notifs from others later.
4. Reader banner campaigns
   • The campaigns I have in mind and the ones that have been run do not share that much in common. The 2022 banner had a simple b&w image, 50 words of text, and then in smaller font 'create an account an edit', which took people to the long, wordy, confusing account-creation page (which I still today sometimes bounce away from when I'm trying to create an account for someone in a rush and the first attempt fails for some reason and we decide just make an anonymous edit).

   Thank you to our donors, editors, and readers!
   Wikipedia is edited entirely by volunteers, and supported by reader donations. You can help Wikipedia grow by learning how to edit today. Many people start with something as simple as fixing a spelling mistake or adding some missing information to an existing article.
   Create an account and edit →

   Colorful, bountiful, brief banners in other contexts can double click rates, account creation could be automatic (temp account), first edits should be on rails (could even be reliving an existing edit), and the new editor should get an initial welcome similar to the welcome templates organic new accounts get. That could get us from 1 in 100,000 activations to 1 in 1,000 without new reader segmentation, client-side / differential-privacy session tracking, or novel discoveries [improvements to wording] from A/B results.

   (Newbies might also be offered a choice to watch a tutorial, join an editathon, or just start editing)

   However, having additional reader segmentation like "clicked on some other banner" or "made a donation" or "visited a project/talk/history page" or even "visited an edit page", or "read about category X" could give further improvements -- if that data is available to people who can craft and test messages and pageflows, and if there's a way to target banners to those pageviews.

5. Other groups interested in this

   There are many networks of people who care about newbie recruitment, engagement, and retention. Starting with those in the Teahouse and equivalent, those maintaining and applying welcome templates, and those manning the Reference Desk / watching those templates.

Sj (talk) 20:23, 28 March 2024 (UTC) (Clarifications on 00:05, 4 April 2024 (UTC))

@Sj Thanks for all the feedback! Some initial thoughts:

1. A/B testing: WMF is investing in a Metrics Platform to help improve our testing capabilities. And that work is prioritized in the next WMF Product & Technology annual plan (SDS2.1). I'm so glad this work is being prioritized!

   Beautiful. :)

2. Giving people an immediate way to edit after clicking a "Try editing" button: I agree that there is more we can do to streamline this process, but I do worry that if we provide an immediate way to complete very easy edits that we would receive patroller pushback. One of the reasons we've been slow to release "add a link" to English Wikipedia, is that patrollers on a few larger wikis have complained that the resulting edits are "low value" and an effort to patrol. We wanted to address some of their complaints before scaling the task out to English and German Wikipedia. The task is Community Configurable, so admins can decide to disable the task. I'm concerned that if the initial launch on English Wikipedia doesn't go well, the task may never be enabled again. That being said, I think we are ready to release "add a link" to English Wikipedia, and @Trizek (WMF) will start communication with English Wikipedia about releasing "add a link" ASAP (T354395)!

   I would counter that it's always better to give more people the experience of completing edits quickly and warmly, and to separate that from the question of how visibly or permanently their edits are propagated, with what sort of patrolling by which combination of reviewers and classifiers. Other levers for streamlining patrolling include showing the same edit to multiple newbies, giving newbies a review task so they are patrolling edits of other newbies (recapitulating the origins of group wikis, in a self-scaling way :), and holding a set of edits without applying them to articles, while gathering enough editor context for a classifier to have a good chance of estimating edit quality. Sj (talk) 00:05, 4 April 2024 (UTC)

I love the sentiment that "it's always better to give more people the experience of completing edits quickly and warmly".

I think the Growth team started with this approach, but then realized that we can't ignore the patrolling side of the equation since ultimately more edits from newcomers means more effort needed reviewing those edits. But you are right, there are ways we could consider somehow bringing newcomers into the reviewing/patrolling side of the equation. We've actually discussed a few of the ideas you mentioned, and perhaps it's time we revisit these ideas as we are discussing investing more in these easy structured tasks for newcomers as part of the upcoming annual plan (Growth will focus on WE1.2).

Growth will be working more with the Editing team in the coming year, and although it's not quite what you are suggesting, I think that the Editing team's work on Edit Check is in alignment with this general sentiment: basically when we surface "checks" to editors that appear to be violating a policy, it helps new editors review their own edits more critically and new editors are less likely to publish edits that end up getting reverted. The Editing Team just published a status update about the Edit Check A/B test today.

As we make plans for the coming fiscal year, I'll start to think more about these other levers we have for streamlining patrolling. And I will follow up with you to get feedback if we propose a potential improvement. Thanks again for all of the feedback! Best, KStoller-WMF (talk) 21:48, 4 April 2024 (UTC)

1. Topical reader/donor campaigns: I'll ask someone from Fundraising to review your banner suggestions here. I like the idea that there is more we should do around connecting editors with Wiki Projects and aligning suggestions with their topical interests. It's not exactly what you are suggesting, but the Growth team hopes to complete an experiment in this area soon: Growth/Community Updates. We are early in the planning, designing, and community outreach with this idea, so please chime in on the associated Talk page if you have any feedback!
2. Reader banner campaigns: I'll ask someone from Fundraising to review this feedback as well. (The Growth team helped create the Special Account Creation landing page and analyze the results of this experiment, but we generally aren't involved in Banner campaigns).
3. Other groups interested: Thanks! I'll try to be more proactive about gathering feedback for our future projects. I always post project information and Growth team updates on MediaWiki, but I recognize there is just so much going on within the Wikimedia Movement that I need to be a little more proactive about gathering feedback.

Thanks again for the detailed feedback and input on this experiment (and other Growth projects in the past). While we're making incremental improvements in the right direction, there's still much more we can and should do to effectively onboard and retain new editors! KStoller-WMF (talk) 22:04, 2 April 2024 (UTC)

Thanks again for sharing your process and thinking. A pity things are spread out across so many spaces, perhaps we could use a #hashtag implementation! Sj (talk) 00:05, 4 April 2024 (UTC)

These are really neat ideas, @Sj. Thanks @KStoller-WMF for tagging me in and for the engagement above. #EditorGrowth as a hashtag?

Regardless, we have some key contributors right here, so let me add a bit more from Fundraising on the topics y'all have been discussing.

Regarding Topical reader/donor campaigns

For the reader banner that is a bit personalized based on the article category: this is something we’ve thought about and I would love to explore this idea further with you and others watching this space. We would want to be careful that we strike a good balance of ‘catching the eye’ with appropriate customizations that are more compelling than our standard message, without the reader feeling like we’re peering over their shoulder. Also, for what it’s worth: we’ve tried images in our banners in the past, too, and could test again. Previously we found bigger gains by deemphasizing graphics in favor of more space for powerful messaging.  

I’d suggest we pick a couple categories to focus on, ones with a broad base of articles and pageviews, while also being specific enough to write customized messaging around. Any suggested categories?

Regarding reader banner campaigns

I think you’re right that we would find a lot of luck if we applied an iteration strategy to a given editor recruitment banner campaign. For now, we aren’t planning any Fundraising-led experiments in this, but we will continue to offer the Learn to edit call to action on our Thank You page. We are also building out more testing infrastructure on our Thank You page so that it’ll be easier to test alternative messaging, layouts and graphics for all our featured CTAs; I've posted more details about that below in the "Questions about the 2023 trial" section. - SPatton (WMF) (talk) 18:27, 12 April 2024 (UTC)

@SPatton (WMF) How can we put A/B testing infrastructure (for non-fr banners) in the hands of more editors? That seems like the critical breakthrough for discovering new things.

For topics: it's worth pinging people tracking interest groups (on en:wp, see the wikiproject council). I would start with the largest wikiprojects by size and activity. I don't have recent stats on hand [we should generate them...] but something for animals, sport, and current events? The fantastic Tree of Life is surely still near the top (how many wikiprojects have their own sibling Project?) and current events portals and workflows have a bit of custom tooling and dedicated community spaces on larger wikis.

In general, I (and perhaps others) would feel better about topically custom banners for other purposes (fundraising, other nudges) if we had a steady flow of their uses to recognize / amplify / improve knowledge on the projects in that area. That way it doesn't feel disingenuous to specifically ask for support for category X when there's no proportionate energy going to improving that category. For #EditorOutreach, Sj (talk) 14:40, 14 April 2024 (UTC)

Questions about the 2023 trial and statistics

• Do you have data on how many clicked through on "create an account" vs the other 4 buttons, like "shop wikipedia"?  
• Is there a place to see the variations tried for each call to action, text and icons?
• Is there a mechanism for running very-small-scale experiments on similar messages during the year?

There are newbie recruitment campaigns in many languages and regions; it would also help to see more of an overview of what's been tried over time. Other wikis (and non-wikimedia wikis) have also had their share of success stories with various recruitment drives, mainly through messsaging on their own sites; that may be more of a lit-review and research question.

Sj (talk) 20:27, 28 March 2024 (UTC)

Thanks for feedback and questions, @Sj! I'll follow up with my colleagues and try to get answers to your questions. KStoller-WMF (talk) 03:52, 2 April 2024 (UTC)

Yes, thanks @Sj! I was out of office a bit last week but wanted to acknowledge that @KStoller-WMF flagged these interesting questions to me and I'll follow up with some more info on the fundraising side this week. - SPatton (WMF) (talk) 20:37, 9 April 2024 (UTC)

Hi @Sj, adding some additional details here in reply to your three bullet-points above.

• We don’t currently track clicks on our Thank You page so we have limited data on click distribution among these CTAs. We know that our primary Call to Action at the top of the page drives many clicks.
• We’re building out more instrumentation on the Thank You page right now so that we can both customize its content more easily for each donor, as well as get richer data back around how people are currently using it.
• We didn’t try other variations of the Try Editing CTA, partly because of the points above indicating our relatively simple page setup right now. I see your many good suggestions above re: how to present this CTA more effectively and would be happy to make space in the new fiscal year for better testing here.
• And yes, there is a mechanism for running small scale experiments on this Thank You page. As @KStoller-WMF mentioned, it would take a bit of coordination with Product on tracking, but is feasible.

- SPatton (WMF) (talk) 18:30, 12 April 2024 (UTC)