Two weeks after releasing the first iteration of "add an image" to Czech, Bengali, and Arabic Wikipedias on November 29, we calculated the "leading indicators". These are metrics that help us quickly understand whether a new feature is behaving in a healthy way. Overall, the most notable thing is how few edits have been completed so far: only 89 edits over the first two weeks. Over the first two weeks of "add a link", over 1,000 edits were made. That feature was deployed to both desktop and mobile users, but that alone is not enough to make up the difference. The leading indicators give some clues. For instance, task completion rate is notably low. We also notice that people do not do many of these tasks in a row, whereas with "add a link", users do dozens in a row. This is a prime area for future investigation.
Mockup of "add an image" feature, showing the caption step for desktop.In terms of development, the team is now working on the desktop version of the workflow. We expect to build it quickly because its design is quite similar to mobile. We'll build an initial desktop experience and do user tests with it in January amongst Spanish and Arabic users. Then we will make changes based on the results and deploy to users. A screen from the desktop design is in the accompanying image. To see the full set of designs, visit this Figma file.
In collaboration with Wikimedia Argentina, we are planning a GLAM event in March 2022. Museum professional will intensively use "add an image" to illustrate articles in Spanish Wikipedia. This will help event attendees get involved with editing, it will improve the wiki, and it will generate valuable feedback about the experience so that we can improve the feature. Right now, we are in planning stages of the event, discussing the topics to which the participants would narrow their suggested edits feeds.
We released the first iteration of "add an image" to Czech, Bengali, and Arabic Wikipedias on November 29.
It is currently being given as the default suggested edit task type to 40% of accounts created on mobile devices (the feature is only functional on mobile). 40% are receiving the Growth features without the new "add an image" task, and 20% remain a control group without the Growth features at all. This experiment will teach us whether "add an image" impacts newcomer outcomes.
So far, we are seeing fewer edits than we expected. Though a relatively small number of users have the feature, it is noteworthy that only about 50 of these edits have been made by newcomers in the first 10 days of its deployment.
The edits that have been happening, though, generally seem to be of the level of quality we were expecting -- most are productive image additions, and most of those have valid captions. But we are also seeing many problematic captions.
Next week, we will conduct our first data analysis of the edits and begin to draw our first learnings.
As a next step, the team will be building the desktop version of the task, for likely deployment in January or February.
The first iteration of this structured task was deployed on May 29, and it is currently on ten wikis. We have learned a lot, and we are now planning a set of improvements for its second iteration. We have learned two important things:
The feature increases engagement from newcomers (we'll be posting data about this in January).
There are clear places for improvement as identified by community members in Arabic, Dutch, Hungarian, and German Wikipedias.
To prepare for the second iteration, we are running data analyses and asking for community thoughts -- any ideas, questions, or concerns. Our ambassadors will be asking for feedback in local languages on Czech, Arabic, Bengali, and Spanish Wikipedias. We hope that other community members with thoughts post them on the talk page!
Update 2021-11-23: upcoming release of "add an image"
We are planning to release the first iteration of "add an image" to Czech, Bengali, and Arabic Wikipedias on November 29.
The experience is currently being tested by the native-speaker ambassadors in those three Wikipedias, and no blocking bugs have been found.
This is an ambitious project that we know contains risks and open questions, which is why we're deploying it carefully and only to a small number of wikis to begin.
After two weeks, we will look at initial data and initial community reactions to decide if the feature is behaving well. We may then decide to expand it to more interested Wikipedia communities.
Scaling
English Wikipedia's test of the Growth features with 25% of newcomers is now complete, with results posted here. The community is discussing whether to increase to full participation.
Our designs and specifications are now finalized, and they all exist as Phabricator tasks here.Mockup of "quality gate" moment when user completes all their image suggestions for the day
We have decided to include a "quality gate": we will be limiting users to 25 image suggestions per day, in order to prevent a confused or bad faith user from damaging many articles quickly. We'll also be using it as an opportunity to encourage the use to try other types of edits. This idea came from the Arabic and German communities as their wikis experienced "add a link". If it goes well for "add an image", we'll likely want to go back and apply it to "add a link", too. You can see the moment when the user hits their limit in the accompanying mockup.
Scaling
Dutch and German Wikipedias have been increased to full participation with the Growth features, with 80% of newcomers receiving them.
Italian, Gan, Inuktitut, and Tajik Wikipedias have the features available for testing. Once these Wikipedias have full deployment of the features, the only remaining Wikipedia without them will be Chinese Wikipedia, which is blocked for technical reasons.
English Wikipedia continues in a test with 25% of newcomers, to be completed on October 20.
GettingStarted was a feature developed in 2013, which directed newcomers to articles that needed editing. We recently undeployed this feature from all wikis, because it has been superseded by the Growth features.
We have posted the findings from our user tests, along with our final designs for Iteration. Our main findings include:
Concept B generally performed better than Concept A, by helping users understand the task better and view more metadata for deciding whether to add the image and how to write the caption.
Our first user tests in a non-English language went well and provided valuable perspective. For instance, we saw that users of Spanish Wikipedia are highly aware of English Wikipedia and tended to defer to its judgment when an image was used on an article there.Screenshot of mentor dashboard on desktop
We developed new ideas for how to help users write smarter captions and automatically validate that their captions seem valid.
Next, the team will continue to engineer on the frontend of the feature informed by these findings.
Today, we deployed the first version of the mentor dashboard to our pilot wikis: Arabic, Czech, and Bengali Wikipedias. Mentors will be able to navigate to the dashboard with a new link along the top of their desktop browser.
This is the most requested feature by mentors, and allows them to see the list of their mentees along with some vital statistics about their progress.
After the pilot wikis use the dashboard for about four weeks, we will plan to deploy to all Wikipedias.
Update 2021-09-27: Growth feature now on nearly all Wikipedias!
Today, we deployed the Growth features to 208 small Wikipedias, which constitute the "long tail" of Wikipedias with low numbers of users.
With these deployments, the Growth features are on over 300 Wikipedias. The only outstanding Wikipedias are:
Wikis that do not yet use Visual Editor (including Chinese)
Italian Wikipedia
English, German, and Dutch Wikipedias, which are all testing the features with small portions of their newcomers.
We are now moving into an era in which Growth features are the standard newcomer experience in Wikipedia -- they are no longer an experiment. This has been three years in the making, and represents a major step forward for new editors.
Going forward, we are going to begin to consider how Growth features should work on non-Wikipedias, particularly Wikisource. We want to hear from community members on how that might work, and conversations have started here.
In the last few weeks, the Arabic and German communities have been having discussions about "add a link" and how it could be improved.
While we make improvements, the German community opted to stop giving the feature to newcomers. This outcome makes sense for their community -- we have a first iteration of the feature, we've learned from it, and we'll be able to return with a stronger iteration in the future.
Both communities have come up with similar ideas, including:
Limiting newcomers to a certain number of articles per day, so that if they are doing a bad job with the links, they'll be limited in how much they can damage articles.
Limiting each article to having only a few suggestions (say, three). Right now, articles can have up to ten suggestions, but this may cause the newcomer to overlink.
Limit suggestions to only articles that appear underlinked, perhaps by looking at the ratio of links to bytes in an article.
Don't suggest links in sections that usually shouldn't have them, such as the References section.
The mentor dashboard is a feature requested by many communities: it is a special page that will allow each mentor to see their list of mentees and how those mentees are doing. With it, mentors will be able to proactively reach out to the newcomers who need their attention most.
We are nearing the deployment of this feature to our pilot wikis. After about four weeks of usage by the pilot wikis, we'll make improvements and plan to deploy to the rest of the Wikipedias.
We hope community members comment on the project's talk page with ideas or questions, or just if they would like the Growth team to prioritize deploying it to their Wikipedia.
Scaling
Following a Village Pump decision, English Wikipedia is now giving Growth features to 25% of new accounts, and mentorship only to 5% of new accounts (so as not to overwhelm mentors). After one month, we will post data from this stage of the trial and discuss next steps with community members.
This week, we'll be deploying Growth features to all the rest of small Wikipedias, which amounts to 208 Wikipedias. After that deployment, Growth features will be on all Wikipedias except Italian, Serbo-Croatian, and a handful that do not have Visual Editor out of beta (including Chinese Wikipedia).
The team is engineering on both the backend and frontend of this workflow.
All our tasks are created, though several are still placeholders as we finalize designs based on user tests.
A couple of interesting tasks include:
Quality gates: how might we stop reckless users from quickly saving many erroneous images onto articles? Our current theory is to cap all users at 25 "add an image" editws per day. In the future, we might do more sophisticated things, like look at the amount of time the user spends on suggestion.
Edit rules: how should we automatically place the image in the actual wikitext of the article? Since we're only illustrating articles that have no images and no infobox, we think this simple rule will work: images should be inserted as thumbnails with captions, under any templates at the top of the article but above the beginning of the actual content.
Scaling
The Village Pump discussion on English Wikipedia signed off on expanding the trial of Growth features to 25% of newcomers. We'll make that change on September 20. In the meantime, we're hoping more English Wikipedians can sign up as mentors.
Next week, we'll be deploying Growth features to all the rest of small Wikipedias, which amounts to 208 Wikipedias. After that deployment, Growth features will be on all Wikipedias except Italian, Serbo-Croatian, and a handful that do not have Visual Editor out of beta (including Chinese Wikipedia).
Update 2021-09-08: discussion on English Village Pump
The team is engineering on both the backend and frontend of this workflow.
Engineering is going significantly faster than it did for "add a link", because we're able to re-use and adapt much of the code we wrote for that workflow.
We're still going through user test results, and we'll post our findings within about two weeks.
We have also posted the Spanish versions of the prototypes to Spanish Wikipedia so that community members can try them out and comment. We hope to hear from many Spanish volunteers!
Scaling
A Village Pump discussion is ongoing on English Wikipedia about increasing the share of newcomers getting the Growth features. We hope to hear from more English community members there!
German and Dutch Wikipedias began testing Growth features last week, including "add a link".
This week, we'll be deploying Growth features to 43 small Wikipedias, which have had the features available for testing.
Update 2021-08-30: processing "add an image" user tests
Last week, we generated 20 user tests with newcomers of the "add an image" prototypes in English, and 10 tests in Spanish.
We're still watching and taking notes on the tests, but we do have some initial takeaways:
Testers seem to understand the task well, including the adding of the caption.
Several elements of "Concept B" worked better than "Concept A" at making the objective of the task clear.
When users saw that an image was used on the same article in other Wikipedias, it increased their confidence in the match.
The team is now engineering on both the backend and starting the frontend of the experience.
Scaling
We're still planning the discussion on English Wikipedia's Village Pump about expanding the set of newcomers receiving Growth features to 25% of new accounts. It will be posted this week.
German and Dutch Wikipedias will likely begin tests of the Growth features this week or next week.
We deployed to 10 small Wikipedias last week: Kurdish, Azerbaijani, Finnish, Lithuanian, Estonian, Marathi, Slovenian, Malayalam, and Georgian.
Next, we will deploy to 43 small Wikipedias listed in this task. Those wikis all have the features available for testing now, to be given to all newcomers next week.
Update 2021-08-23: "add an image" prototypes available for testing
Earlier this month, we posted plans and designs for Iteration 1 and began community discussion.
This week, we've posted interactive prototypes of our designs, which make it easy for community members to try out the experiences we're considering. These designs are not real wiki software, but they work well enough to get through a simulation of the experiences we're planning.
We'll be using the prototypes for user tests with newcomers this week on usertesting.com. For the first time, the Growth team will be running user tests in multiple languages: English and Spanish. We hope community members can check out the prototypes below and leave any thoughts on the project talk page:
Our user test results will allow us to make decisions about the exact experience we'll implement for Iteration 1, and we'll post the notes from the tests once we have them assembled.
In the meantime, the Growth team is starting to build the backend of the feature.
Scaling
This week, we'll start a discussion on English Wikipedia's Village Pump about expanding the set of newcomers receiving Growth features to 25% of new accounts.
German and Dutch Wikipedias are still planning to begin testing Growth features in September.
We'll deploy the Growth features to newcomers on 10 small Wikipedias this week: Kurdish, Azerbaijani, Finnish, Lithuanian, Estonian, Marathi, Slovenian, Malayalam, and Georgian.
The Growth team will be hosting a table in the Community Village on the last day of Wikimania. We will be at Building 1, Floor 1, Table B from 14:30 UTC to 16:00 UTC. Community members can visit us to talk about any of our work, see our latest designs, or ask questions.
The team's work was represented at three sessions so far, with videos available on YouTube:
"Editing the Wiki Way: Software and the Future of Editing": a discussion about how our editing software can change to make editors more successful in doing difficult tasks. We'll saw demos from community members who have built innovative editing tools to spur the discussion.
"Newcomer Experience and Content Campaigns": this was a lightning talk about how two teams at WMF (Growth and Campaigns) will be working across WMF departments to think about newcomers and campaigns beyond just our software.
We've posted our plans and designs for Iteration 1 to English, Arabic, Vietnamese, Czech, and Bengali audiences, and we have started to receive community thoughts.
We hope community members can continue to weigh in our design concepts and help us anticipate issues we'll have while building this iteration.
As we refine and user tests our designs, we're starting work on the backend for the feature.
Scaling
The Growth features are now live on 39 Wikipedias, including most of the large and medium wikis.
English Wikipedia is in the process of experimenting with the features among a small percentage of new accounts.
German and Dutch Wikipedias will be starting similar tests in September.
We are now moving on to deploying the features to the many smaller Wikipedias, starting with a group of ten that currently have the features available for testing before going broadly to newcomers: Kurdish, Azerbaijani, Finnish, Lithuanian, Estonian, Marathi, Slovenian, Malayalam, and Georgian.
Going forward, we'll deploy to the rest of all Wikipedias in large groups over the coming couple months.
Update 2021-08-10: designs for Iteration 1 of "add an image"
Wikimania is coming up later this week weeks, and the Growth team's work is going to be part of four sessions. We hope that community members can attend:
"Editing the Wiki Way: Software and the Future of Editing": this will be on August 14th at 17:45 UTC. It will be a discussion about how our editing software can change to make editors more successful in doing difficult tasks. We'll be hearing from community members who have built innovative editing tools to spur the discussion.
"Newcomer Experience and Content Campaigns": this will be a lightning talk on August 15th at 21:20 UTC. We'll be talking about how two teams at WMF (Growth and Campaigns) will be working across WMF departments to think about newcomers and campaigns beyond just our software.
Growth team community village table: this will be on August 17th at 14:30 UTC. This is like office hours for the Growth team, where anyone can come by to talk about our work, plans, data, designs, or to bring us ideas and questions!
We have posted our plans and designs for Iteration 1 of the "add an image" work. We'll be working on this second structured task over the next three months.
We hope community members can look over what we're planning, consider our different design concepts, and comment on the talk page. We want to incorporate community thoughts into our decisions. We are also posting these plans on Arabic, Czech, Vietnamese, and Bengali Wikipedias.
As we gather community thoughts, we'll also be running user tests of our design concepts. Then we'll use all the information together to decide what exactly to build for our first iteration.
Wikimania is coming up in less than two weeks, and the Growth team's work is going to be part of at least two sessions. We hope that community members can attend:
"Editing the Wiki Way: Software and the Future of Editing": this will be on August 14th at 17:45 UTC. It will be a discussion about how our editing software can change to make editors more successful in doing difficult tasks. We'll be hearing from community members who have built innovative editing tools to spur the discussion.
"Newcomer Experience and Content Campaigns": this will be a lightning talk on August 15th at 21:20 UTC. We'll be talking about how two teams at WMF (Growth and Campaigns) will be working across WMF departments to think about newcomers and campaigns beyond just our software.
The team is working on mockups for how this task might look for users. This week, we'll post the ideas we're considering so that community members can discuss.
Next, we'll run user tests with newcomers to understand which designs will work best.
Over the past weeks, the team has been working on several improvements to add a link.
Edit mode toggle: this will allow newcomers to switch over to visual editor while doing the "add a link" task, which means they can pursue more complicated edits if they see the opportunity while doing the task.
Mobile swiping: this will make it easier and more fun for mobile users to choose a task they want to do.
Hide link inspector: for mobile devices with small screens, the user will be able to temporarily hide the link inspector to see more of the article.
Update 2021-07-26: "add a link" leading indicators
Last Monday, July 19, we sent emails to 4,479 donors from Spanish Wikipedia and 1,108 donors from Portuguese Wikipedia asking them to create accounts to edit. These donors had previously indicated on a post-donation survey that they were interested in learning to edit.
The email send went well -- we received no complaints either from recipients or communities, and our software for receiving these accounts worked as expected.
So far, 136 accounts have been created in Spanish Wikipedia and 35 in Portuguese Wikipedia, resulting in conversion rates of about 3%. About a fifth of these accounts have made edits.
In about two weeks, we'll analyze the data and compare to baseline to understand the value of recruiting donors as editors in the future.
We published our "leading indicators" for this feature, which are the numbers we calculated two weeks after deploying "add a link" to understand if behavior looked healthy and if there were any clear problems.
We're seeing strong results so far from "add a link".
Newcomer edits made through the feature had a significantly lower revert rate than edits made without it.
Newcomers with the "add a link" feature completed many more suggested edits than those without, suggesting that the feature encourages activity.
Newcomers accepted about two thirds of the suggestions, indicating that in their judgment, the algorithm is mostly correct, but makes enough mistakes that their involvement is valuable.
We are not seeing many newcomers indiscriminately answer "yes" to all suggestions, a negative behavior that we feared.
An area for improvement is "task completion rate", which is only 54%, meaning that almost half of users who start an "add a link" task don't complete one. This is lower than expected.
After seeing these positive results, we deployed "add a link" to six more Wikipedias: Russian, Polish, Romanian, French, Persian, and Hungarian.
Last Wednesday (2021-07-14), we sent emails to 100 Spanish-speaking donors and 100 Portuguese-speaking donors to test our workflow. We saw some users successfully create accounts, and so we knew the workflow was behaving correctly.
We also announced the upcoming email on the village pumps on those two wikis.
Because the test was successful, today, we sent emails to full list of about 4,500 Spanish-speaking donors and 1,100 Portuguese-speaking donors. So far, we're seeing dozens of accounts getting created, and all is going according to plan.
In the coming weeks, we'll analyze how users behaved in this workflow, looking for whether these donors created accounts at a high rate and became high-value newcomers.
So far, the Growth team has deployed one structured task ("add a link") and has started work on a second one ("add an image"). Both of these tasks rely on algorithms that required months of background research, adjustment, and validation.
When we originally discussed structured tasks with community members, they made it clear that it would be very valuable to have some sort of task around copyediting, in which newcomers could address spelling and grammar issues. These are notes from those community conversations.
To that end, we've started doing background research around what might be possible for a future task around copyediting, and began a project page to track that research.
We encourage community members to check out how we're thinking about this, and to chime in on the talk page with any ideas or concerns!
Update 2021-07-12: fundraising experiment on schedule
This experiment is still on schedule for sending emails to Latin American donors who have expressed an interest in editing:
We plan to send a test round of emails to a small set of targets on July 14.
We plan to send the full flight of emails on July 19.
We have created special account creation pages to be the "landing pages" for these emails (i.e. the pages that load when a user clicks the "create your account" button in the email. These pages are meant to show the newcomers that they're still on the right track.
The team has been communicating with Latin American affiliates about the upcoming email, and we'll be posting on Spanish and Portuguese village pumps this week.
After analyzing results from the Android MVP, we have decided to build a first iteration of this structured task during the coming three months.
This is an important decision, and we'll be posting more information about the decision this week. In short, we think that the usage of the Android MVP shows that:
Newcomers will have high engagement with the task, meaning that they will be interested and motivated to do it.
Newcomers will have high efficacy with the task, meaning that the accuracy of the image suggestion algorithm combined with their judgment will yield good edits.
We still have many open questions about whether and how this task will work well, and that's why we are only committing to a first iteration. We'll start community conversation about it in the coming weeks.
Scaling
English: this week, we'll summarize results from the English Wikipedia test so that community members can discuss next steps.
Dutch: this Wikipedia now has the Growth features available for community members to test.
Update 2021-06-28: "community configuration" deployed to Growth wikis
After about a month of being deployed on Arabic, Czech, Vietnamese, and Bengali Wikipedia, we have deployed "community configuration" to all wikis that have Growth features.
This capability allows administrators to configure Growth features for their wikis without intervention by the Growth team, all through a user-friendly form.
Administrators can change help links, set templates for suggested edits, and change settings for mentorship features as experienced by all newcomers. Changes are all tracked as page history.
To access the form, administrators can go to Special:EditGrowthConfig on their wikis.
We think that this will allow communities to be empowered to tailor Growth features to better fit their own wiki culture and needs.
Account creation with Fundraising
This experiment now has a clear timeline, in which we'll email donors in Latin America who indicated an interest in editing, and ask them to create accounts to edit.:
We plan to send a test round of emails to a small set of targets on July 14.
We plan to send the full flight of emails on July 19.
Team members will be communicating with Latin American wikis and affiliates to make sure that everyone is aware of this experiment.
Because of the comparatively small number of email recipients (several thousand), we do not expect dramatic impacts on the wikis, but we do expect to learn whether and how donors can be converted into editors. We'll use this knowledge in planning future tests along these lines.
Scaling
This is the fourth and final week of the 2% test in English Wikipedia. During the week of July 12, we'll summarize the results for community members to discuss next steps.
We have seen positive indicators that "add a link" is working well in our four pilot wikis. For that reason, we're preparing to deploy it to about eight more wikis in the coming weeks. If we continue to see positive results, we'll then deploy to all Growth wikis.
Update 2021-06-21: moving forward with Fundraising experiment
We're continuing to develop this planned experiment in which we'll email donors who indicated an interest in editing, and ask them to create accounts to edit. This will take place amongst donors to a recent Latin American fundraising campaign.
We've established that donors will receive the email, and after they click the link, will land on a special version of the Create Account page, acknowledging their recent donation and indicating that they're on the right track. That's being designed here.
After creating their account, the donors will skip the welcome survey and go straight to the homepage. This will get them more quickly into the editing experience, and we know we won't be interrupting any in-progress edits (because they just arrived from an email).
This test will likely happen in mid-July, and we'll be looking to learn whether many donors do follow through and start editing.
Scaling
This is the third week of the 2% test in English Wikipedia, with throughput still holding constant at about ten suggested edits per day. After next week, we'll summarize the results from the test and discuss next steps with the community.
We'll be deploying Growth features to Latvian and Slovak Wikipedias this week.
Right now, the work of the Growth team begins when a user creates their account. That's when the Growth features start to invite them to their homepage to begin editing.
Going forward, the team is going to start thinking about how to encourage more users to create accounts, beginning that process. Now that the Growth features are a safe place for newcomers to get started, we feel comfortable starting to invite more people to become newcomers.
As a first experiment, we're working with the WMF Fundraising team on an idea to encourage donors to become editors. In a recent Latin American fundraising campaign, donors were asked in a post-donation survey whether they were interested in learning about editing. Our idea is to email those who said "yes", to encourage them to create accounts and get started.
We'll be communicating with the Spanish Wikipedia community about this idea as we develop it.
We are now in the third week of this feature being deployed, and things are still looking good. We're spending time cleaning up bugs and making small leftover improvements to the feature that we did not complete before its release.
This week, we'll generate our "leading indicator" statistics, which will be our first clear glimpse of important metrics like revert rate. We'll use those to decide when and how much to extend the feature to more wikis.
If the leading indicators are in the right range, we'll talk to communities about deploying "add a link" to these Wikipedias next: Russian, French, Polish, Romanian, Persian, Hungarian, Spanish.
Scaling
This is the second week of the 2% test in English Wikipedia. We're seeing newcomers make about ten suggested edits per day, which is about the rate we expected.
After two more weeks, we'll assess the results and discuss with the English community on next steps.
Update 2021-06-07: performance of "add a link" so far
Since we deployed it on May 27 (12 days ago), users have made "add a link" edits to 844 articles across our four pilot wikis (Arabic, Czech, Vietnamese, Bengali).
The revert rate appears to be quite low for these edits (less than 10%).
Some notes we've collected from observing the feature so far:
An enthusiastic newcomer made hundreds of "add a link" edits on Arabic Wikipedia, but started out accepting all suggestions until another user reminded them to evaluate them more critically. This taught us that the feature could automatically intervene if a user seems to be accepting too many suggestions, which we are discussing in this Phabricator task.
Patrollers may not know about the feature and it's automatic edit summary. We've seen an instance of a user who thought that the automatic edit summary (e.g. "Link suggestions: 3 accepted, 2 rejected, 0 skipped") was written by hand by the newcomer. We're considering how to clarify the message in this Phabricator task.
Next week, we'll collect statistics from this deployment to understand what improvements are needed and to decide when and how to deploy the feature to more wikis. This Phabricator task lists the wikis to which we're considering deploying it next.
Scaling
We will start a test with the English Wikipedia community tomorrow on June 8, in which 2% of newcomers will receive the Growth features.
The conversation about the test is happening here, for community members to join in.
Update 2021-06-01: "add a link" and "community configuration" releases
Add a linkScreenshot of the "add a link" feature in on a mobile device
We deployed the "add a link" feature to our four pilot Wikipedias on May 27 (Arabic, Czech, Vietnamese, Bengali).
Half of newcomers have "add a link" available as their default task and the other half still have the legacy task type based on maintenance templates. This is an A/B test to see how the new feature impacts newcomer engagement.
So far, there have been over 500 edits made through the new feature.
In about two weeks, we'll analyze the initial data and determine changes to make, and whether to scale to more Wikipedias.
We've looked at the initial data from the Android MVP, and it looks encouraging! We're seeing hundreds of people evaluating thousands of images, with most people doing many images per session. We hope this means that the task is interesting.
In the coming weeks, we'll be evaluating the data closely to understand how engaging the task, how strong the judgment of the users is, and how well the task works across languages. This will help the Growth team decide if and when to build it for the web.
Scaling
Wikipedias that started receiving the features this week are: Urdu, Bulgarian.
We will soon start a test with the English Wikipedia community on 2% of newcomers. Experienced editors have tried out the features, and we are deciding when to begin the test. Follow along here.
Update 2021-05-10: "add a link" nearing deployment
While the Growth team has not been working on this effort recently, the Android team has.
Last week, they deployed what we're calling the Android MVP, which collects image annotations from users without saving any edits.
Over the coming weeks, we'll learn a lot about how users like this task, whether they apply good judgment, and how well it works across languages. These learnings will help the Growth team decide whether to build such a task for the web.
Scaling
English Wikipedia community members are testing out the features. They have nine mentors signed up, and so far, the testing is going well. Follow along here.
Wikipedias that will start receiving the features this week are: Catalan, Greek.
Wikipedias preparing for deployment are: Urdu, Bulgarian, and Latvian.
"Add a link"A screenshot of "add a link" feature in development in beta.
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
The experience is sufficiently developed that community members are welcome to try it out on Test Wikipedia. It still contains many visual bugs that we're fixing.
Scaling
Wikipedias that will start receiving the features this week are: Catalan, Urdu, Greek.
Wikipedias preparing for deployment are: Bulgarian and Latvian.
The team has started a project to help more communities get set up quickly with Growth features, and to allow them to adjust the features on their own.
In this idea, administrators will be able to use a Special page on their wikis to adjust the configuration of the Growth features for all users. For instance, they'll be able to change which templates put articles into the newcomer tasks feed.
While this idea was developed for the Growth features, it has applicability for other features, and so we hope community members read through the project page and offer thoughts on its talk page.
"Add a link"Mockup of mobile version of mentor dashboard
We decided to use the term "machine" to refer to the link recommendation algorithm in the feature (as opposed to the term "artificial intelligence"). To learn about how we made that decision, see this section.
We posted our plans for measurement and experimentation with "add a link". We'll be testing the feature to see if newcomers have increased engagement with "add a link" than without it. For details, see this section.
We are finishing designs for the mobile version of the dashboard. Though we believe that most mentorship currently happens from desktop users, we believe that making mentorship easy from mobile devices will encourage new kinds of people to become mentors.
Scaling
Last week, these Wikipedias started receiving Growth features: Malay, Simple English, and Tamil.
Wikipedias that will start receiving the features this week are: Spanish.
Wikipedias preparing for deployment are: Serbo-Croatian, Urdu, and Latvian.
Update 2021-04-05: link recommendations algorithm remains accurate
Last week, we tested the link recommendation API outputs in six languages (English, French, Vietnamese, Bengali, Czech, and Arabic) to ensure that the algorithm remains accurate and to find any issues. We found an accuracy level between 70% and 90% depending on the language (we discovered fixable issues in Vietnamese that made its suggestions much lower), meaning that 70-90% of suggested links should be links, and will link to the right article. We believe this is sufficiently accurate for newcomers to be successful.
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
The link recommendations are soon going to be available via public API. The Growth team's "add a link" feature will use this API, but community members will also be able to use it to build their own tools using link recommendations.
This week, we are testing the API outputs in six languages (English, French, Vietnamese, Bengali, Czech, and Arabic) to ensure that the algorithm remains accurate and to find any issues.
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
We are finalizing designs for dashboard after receiving community thoughts, and engineering will begin soon. You can follow along with engineering under this Phabricator task.
Scaling
Last week, Hindi, Japanese, Telugu, Albanian, Esperanto, and Norwegian Wikipedias started receiving Growth features.
Wikis that have the features available for testing are: Spanish, Tamil, Malay, and Simple English.
Wikis preparing for deployment are: Serbo-Croatian and Latvian.
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
Mentor dashboardA mockup showing one potential arrangement of the mentor dashboard
Following user interviews, we have now created an initial set of designs and posted them on the project page.
Community members are encouraged to look at the designs and give us their thoughts by answering the questions linked from the page. We'll also be going directly to the mentors on our pilot wikis for their thoughts.
In the next two weeks, we'll make final designs and begin engineering.
Scaling
Last week, Indonesian and Croatian Wikipedias started receiving Growth features.
Wikis that have the features available for testing are: Japanese, Esperanto, Norwegian, Albanian, Hindi, Malay, and Simple English Wikipedias.
We are now working on a timeline that targets deployment in early April. We'll first deploy to our pilot wikis: Czech, Vietnamese, Arabic, and Bengali Wikipedias. After two weeks, we'll look at these leading indicators to see if there are any major problems. If not, then we'll deploy to the other wikis that have the Growth features:
Revert rate: the percentage of "add a link" edits that are reverted.
User rejection rate: the percentage of link suggestions for which users select "no" instead of "yes".
Task completion rate: the percentage of users who start the link recommendations task that also complete it.
Scaling
Last week, Danish and Romanian Wikipedias started receiving Growth features.
Wikis that have the features available for testing are: Thai, Indonesian, and Croatian Wikipedias.
Wikis preparing for deployment are: Hindi, Albanian, and Esperanto Wikipedias.
This week, we'll be finalizing our approach to instrumentation and experimentation for this feature.
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
Scaling
Wikis that are preparing for the Growth features include Danish Wikipedia, Thai Wikipedia, Romanian Wikipedia, Indonesian Wikipedia, Albanian Wikipedia, and Croatian Wikipedia.
We are also preparing to discuss deployment on English Wikipedia. To weigh in on how we'll approach our largest Wikipedia, see this discussion thread.
We've completed writing the copy for the feature, and it is now finalized in all the specifications.
We have also decided what we'll be measuring and are finalizing our plan for running a controlled experiment.
Scaling
Wikis that are preparing for the Growth features include Danish Wikipedia, Thai Wikipedia, Romanian Wikipedia, and Indonesian Wikipedia.
For these wikis (and future ones), we are deploying the Growth features in "dark mode", meaning that no users have them on by default. Then communities have two weeks (or longer, if they request) to test, discuss, translate, and configure the features to fit their wiki. If after two weeks, there are no objections, we turn the features on for newcomers.
To facilitate this process, the team is working on a way for communities to adjust and configure the Growth features on-wiki on their own with a simple form. This will help the Growth team to deploy to more wikis more quickly, since each one won't need individualized preparation with Growth engineers.
Update 2021-02-15: results from second variant test
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
We're also continuing to write the copy for the feature and to develop the measurement plan.
Mobile results showing higher editing numbers for "Variant C"Variant C vs. D
We recently completed the analysis of our second variant test of the newcomer homepage. These tests are meant to try different arrangements of the newcomer homepage to see which ones increase newcomer engagement. Half the newcomers get one variant, and half get the other. Our first variant test was completed in June 2020, leading us to deploy "Variant A" to all users.
This test compared Variants C and D, both of which built on learnings from the first test. Both variants make suggested edits the most prominent part of the newcomer homepage, but differ in how users are onboarded to suggested edits.
We found that both Variants C and D led to higher interactions and edits than Variants A and B. Variant D led to more users editing on desktop and Variant C led to more users editing on mobile.
To take action on these results, we will soon be shifting all users of the desktop experience to Variant D and all users of the mobile experience to Variant C.
Update 2021-02-08: statistics and summaries for "add an image"
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
We're also continuing to write the copy for the feature and to develop the measurement plan, which will explain which data we're measuring and how we'll set up any A/B tests.
In December, we ran 15 user tests with newcomers on the "add an image" idea, using this interactive prototype. The objective was to observe as many user judgments on image matches as possible, so we can understand the strength of user judgment and figure out how to design to increase it.
In summary, we think that these user tests confirm that we could successfully build an "add an image" feature, but it will only work if we design it right. Many of the testers understood the task well, took it seriously, and made good decisions. On the other hand, many other users were confused about the point of the task and made weak decisions -- but for those confused users, it was easy for us to see ways to improve the design to help them succeed.
Another important development for this project is the start of work on the Android MVP (minimum viable product): the Android app team will be building a simple image matching workflow as a "suggested edit" in the app. The MVP will not save any edits. It is only for the purposes of learning how to improve the image matching algorithm and the design, so that when the Growth team works on this idea, we can benefit from the learnings.
Our team has been undergoing some changes as we gain new members and old members transition to other teams. We're expanding, which is good! But it means we'll be spending effort over the next few weeks onboarding new members and helping them get up to speed.
We are continuing to engineer on both the backend of the link recommendation service and the frontend of the user experience.
The beginnings of the experience will start being available in beta wikis this week. We will let volunteers know when the pieces start being ready to try out.
We're now writing all the copy for the feature (the words that users will see).
We're also writing our measurement plan, which defines which data we'll gather and how we'll use it. This will get posted on-wiki for community members.
We've finished gathering our notes on the user tests that we ran around this idea. We'll be posting our findings on-wiki this week so community members can see them.
The Android team is going to be building an MVP (minimum viable product) around image recommendations. It's not going to make any edits to Wikipedia: it's just going to help us learn how users react to an image-related task, and give us a sense of how strong their judgment will be when they do the task. We'll link to more details on this idea once it takes shape.
We are finishing up the final threads on the discussion page for this project. We'll summarize the state of the conversation this week, but please feel free to join in at any time.
We'll also be posting learnings from our 15 user tests on the image recommendation prototype.
On the technical side, the Platform Engineering team is scoping out a "proof of concept" API for serving image recommendations. The Growth and Android teams will be able to use this to try out building an "add an image" feature and get a sense for hw such an API would function.
Mentorship is an important part of the Growth team feature set, and it's the one in which community members participate most, and have the most ideas and questions.
But because our team spends the majority of our resources on newcomer tasks, we have not been able to dedicate time to improvements for mentorship.
The team now has a design intern working with us for the coming months, who will be focused on mentorship features.
Over the last years, the most requested mentorship feature is some sort of page where mentors can see who their mentees are and monitor how they are progressing.
The design intern will be working on building this, using the existing project page as a starting point and the existing community thoughts on its talk page.
If you have opinions or ideas, please speak up on the talk page so we can take them into account.
