Good conversations are happening on the discussion page for this potential future structured task. So far, five community members have joined in. We hope more continue to join in the coming weeks.
We're going to be conducting parallel conversations in local languages on Arabic, Bengali, Vietnamese, and Czech Wikipedias so that we can represent diverse perspectives.
This past week, we ran 15 user tests with newcomers, in which they each used the prototype tool to evaluate 20 proposed image matches from the algorithm. We are learning a lot from these user tests about the strength of the algorithm and the sort of judgment that newcomers apply to evaluating images. We will post our findings in January.
Growth features increased activation (making a first article edit) and constructive activation (making a first unreverted article edit)Data on impact
We have completed an analysis of the newcomer tasks feature, which shows that it successfully increases the engagement of newcomers.
The Growth features, and especially newcomer tasks, increase the likelihood that a newcomer makes their first edit, that they are retained to make future edits, and increases the number of edits they make.
This is great news, and a result we have been looking forward to. While we had suspected that the features were having a positive effect, the analysis of this experiment shows it rigorously.
Now that we have these results, we're confident that all wikis would benefit from the Growth features, and we'll be talking to more wikis about deploying the features.
A summary of data on the impact of Growth features is here, and the full analysis of the experiment is here.
We are continuing to engineer on the backend of the link recommendation service.
We are recording exact specifications for the frontend in this Phabricator task and its subtasks.
"Add an image"
We have started to build a quick tool that will simulate what it might be like to evaluate real image matches from the algorithm that the Research team is developing.
While it won't save any real edits, this tool will allow staff and volunteers to get a sense of whether the image matches are strong enough to give to newcomers to evaluate.
Scaling
Bengali Wikipedia is preparing for deployment of Growth features.
Indonesian Wikipedia is preparing for deployment of Growth features.
Latest newcomer tasks numbersThe number of suggested edits completed by each wiki each week, going from December 2019 to November 2020, with a line for the total.The number of distinct users completing suggested edits by each wiki each week, going from December 2019 to November 2020.
Since the deployment of Variants C and D about four weeks ago, the number of suggested edits and users completing suggested edits have both increased dramatically.
While Variants C and D may be part of the increase, the majority is driven by Persian Wikipedia, which has an exceptionally high number of users making many suggested edits. We are investigating what makes suggested edits so popular in that wiki.
In about two weeks, we'll start to analyze the results from the Variants C and D experiment to see what impact it has and to choose the best variant.
Since its initial deployment in December 2019, 5,219 newcomers have made 37,235 edits through this workflow, which is deployed to 17 Wikipedias.
We are continuing to engineer on the backend of the link recommendation service.
The team met last week to go over desktop designs in detail, and this week, we'll be writing down the resulting specifications in Phabricator.
"Add an image"
This is the next structured task we're planning, which would prompt newcomers to add an image from Commons to an unillustrated Wikipedia article based on an algorithm.
We are thinking about how to make a quick tool that would allow us to see how easy it is for people to accurately add images, and whether people find the task interesting and fun.
Update 2020-11-15: preparing for frontend engineering for "add a link"
We are continuing to engineer on the backend of the link recommendation service.
The team met last week to go over mobile designs in detail.
This week, we will be meeting to go over desktop designs. In a previous update, we mentioned that we were deciding between "Concept A" and "Concept B" for desktop. We've decided to go with Concept A, which is the design that puts the user in the context of the Visual Editor, instead of creating a separate place just for suggested edits. After testing both designs, this decision was made for the same reasons that we decided on Concept A for mobile: Concept A brings the user closer to the editor may help them attempt more advanced types of editing.
After going over the desktop designs in detail with the team this week, we'll create Phabricator tasks and start working on the frontend in earnest.
Update 2020-11-08: backend engineering for "add a link"
Our main engineering work continues to be on the backend, building the link recommendation service. As we work on this, we've been deciding on the answers to questions like:
What are the minimum and maximum numbers of suggestions we want to show users in one article?
How often should an article's link recommendations refreshed?
How might we store when a user rejects a link, and then use that to prevent recommending that same link to other users?
The answers to those technical questions, and others, are listed in this Phabricator task.
This week, the team will be meeting to go over our designs for the mobile user experience in detail, and then we'll commence engineering on the part users will see. To see our latest mobile designs, try this interactive prototype.
The following week, we will meet to go over our designs for the desktop user experience.
Mockup of Concept A for desktop "add a link"Mockup of Concept B for desktop "add a link"
Variants C and D
After two weeks of these variants being deployed, we can already tell they are making a big difference to the numbers.
The number of homepage visitors who interact with the suggested edits module has doubled, which is what we expected.
In coming weeks, we will be able to determine whether that increased interaction is leading to more newcomers completing edits (without increases in reverts).
We have completed a set of user tests for two desktop design concepts. Concept A brings the newcomer to the article, while Concept B allows them to edit right from their homepage. See screenshots both concepts at the right, with details and interactive prototypes here.
We are hoping to hear from community members on their opinions of these concepts! Please visit the talk page.
After we finish analyzing the user tests, we will post our findings.
These were deployed on Monday, October 19. All new accounts now have an equal chance of experiencing Variant C or D, unless they are in the control group.
In about 4 weeks, we will analyze the tests to determine which variant is best.
We are now primarily working on this project. Backend engineering is underway, and we are in the process of incorporating user test results to finalize designs.
Later this week, we'll sketch out timelines for the rest of the project.
"Add an image"
Though we're just beginning our first structured task, we're starting to think about a possible next one: prompting newcomers to add images from Commons to Wikipedia articles, assisted by algorithms.
This structured tasks is more ambitious, but potentially more rewarding for users and more valuable for wikis than adding wikilinks.
We'll post more information in the coming weeks about our thinking here, and start community discussion about it.
We plan to deploy these variants of the newcomer homepage on Monday, October 19.
Updated mockup of onboarding screen for "add a link" on mobile. User tests taught us that onboarding needed to be clearer.After about 5 weeks, we will analyze the data from the tests to determine which variant is helping more newcomers get farther in the suggested edits workflow. Then we'll convert all users to that best variant.
We are running a second set of user tests this week. Our first set tested two different mobile design concepts. With this round of tests, we'll be testing the next version of mobile designs and the first version of desktop designs. After the tests are complete, we'll finalize all designs and plan to engineer them.
We've detailed out the backend engineering work required, which can be seen as subtasks under this Phabricator task. The work is largely about creating a "link recommendation service", which can generated suggested links for any article and store them in an efficient manner.
Though we planned to deploy Variants C and D today, October 13, we have been delayed by issues with the deployment train.
The variants will be deployed on Monday, October 19.
As we transition from engineering on Variants C and D to engineering on "add a link", we'll be spending about two weeks fixing some bugs and doing some maintenance.
Engineering work on the backend for this project is happening in earnest. We are making a service to look up recommended links for any article, which we'll use for the frontend of the feature.
We are also working on the designs for the desktop version of "add a link".
Deployments
Turkish Wikipedia: coming up next.
Update 2020-10-05: "add a link" user test findings
We will deploy these new variants on Tuesday, October 13. We're finishing the final fixes this week.
When deployed, all new accounts will have a 40% chance of being assigned Variant C, 40% chance of being assigned Variant D, and 20% of not having Growth features at all (this is our control group).
We will also be converting all existing account holders who have Growth features over to having Variant D. Users who turn on Growth features in their preferences will also have Variant D.
After about five weeks, we will analyze interactions to determine which variant leads to more unreverted edits, and we will convert all users to the winner.
Next, we are designing and testing for desktop, and making the next set of mobile designs.
Engineering for the backend of this feature has already begun with the help of the Research and Site Reliability teams: we are setting up a service for the link recommendation algorithm.
The Growth team's engineers will turn attention to this project fully in about two weeks.
Deployments
Portuguese Wikipedia: deployed to newcomers after successful trial week.
Swedish Wikipedia: deployed last week.
Turkish Wikipedia: coming up next.
Update 2020-09-27: nearing release on Variants C and D
Variants C and D engineering: all parts of these variants have been coded, and are now being tested and fixed. We expect to deploy in the next two weeks.
After analyzing our 10 user tests, we think we have a decision about whether we want to build Concept A (edits take place in the visual editor) or Concept B (edits take place in their own dedicated space) for mobile.
Users seemed to understand the task more easily in Concept A. This design also has other advantages, many of which are discussed by community members here.
We will post detailed findings from the user tests this week.
Next, we will design and test for desktop.
"Add an image": last week, we evaluated a second version of the algorithm. This second version is meant to have higher accuracy, by focusing on only on images connected to the Wikidata item of an article with no image. Accuracy did improve as expected, and now we have to decide what further improvements to make before using the algorithm for a feature. That work happened on this Phabricator task.
Deployments
Polish Wikipedia: deployed last week
Portuguese Wikipedia: deployed last week in "trial mode". People only have the features if they turn them on in preferences.
Variants C and D engineering: we are still on schedule for releasing these variants in early October. This week, we're testing most of the user experience components and working on the instrumentation.
We're now processing the results from our user tests, and working on decisions around Concept A and Concept B. We'll post the summary of our learnings from the tests in the next week or two.
We've started talking with the Editing team about the engineering work required to build this feature. Since it will involve the Visual Editor, we'll need to work with them and draw on their expertise.
"Add an image": this week, we'll evaluate a second version of the algorithm. This second version is meant to have higher accuracy, by focusing on only on images connected to the Wikidata item of an article with no image. That work is happening on this Phabricator task. This evaluation is meant to help us decide whether the algorithm could be usable in a version of an "add an image" structured task in the Android app.
Upcoming deployments: we plan to deploy Growth features to the following wikis in the next two weeks.
Variants C and D engineering: we are still on schedule for releasing these variants in early October. We expect them to increase the number of newcomers who try suggested edits.
We completed 10 user tests last week, using these prototypes of Concept A and Concept B. This week, we'll be watching the tests and using them to help decide on a final design concept.
An active community discussion on the design concepts for "add a link" is happening at this page. We hope community members continue to join in this week.
"Add an image": we have been planning ahead for a potential future structured task that encourages newcomers to add images from Commons to appropriate articles. The first thing we need to figure out is whether we can have an algorithm strong enough to make good recommendations. Our evaluation of the first draft of this algorithm took place on this Phabricator task. Right now, about one third of its recommendations are good. We see clear areas for improvement and will be working on and evaluating another iteration.
Variants C and D engineering: we have progressed enough in the engineering of these variants that we're starting to talk about the timeline for launching them. We expect this to happen in early October.
Upcoming user tests for "add a link": last week, we posted new mockups of the "add a link" structured task. As we gather community thoughts about the designs, we'll also be gathering reactions from people new to editing Wikipedia through usertesting.com tests. We'll be doing 5 tests of each of the two design concepts. Here is a link to the interactive prototype of one of the concepts.
Screen from a design concept for the "add a link" structured taskNew mockups of "add a link": back in May, we had a good community discussion about the "add a link" structured task (summary here). We then took community thoughts into account as we made new design mockups. We hope community members can check them out and react to them.
Russian Wikipedia: we have deployed the Growth features to Russian Wikipedia, which gets about 12,000 newcomers per month. Next, we're planning a deployment to Polish Wikipedia.
Variants C and D engineering: we are mainly engineering on this project, meant to increase how many newcomers attempt suggested edits.
Meetings on "add a link": this week, the team will have a series of meetings planning for the "add a link" structured task. We'll post in-progress mockups of our ideas during this week so that community members can see and give us advice. We're also simultaneously planning to user test the designs to get thoughts from people new to editing.
"Add an image": although we are designing our first structured task using a link recommendation algorithm, we are also considering a future structured task that would prompt users to add an image from Commons to an article that doesn't have any images. A member of the WMF Research team is working on an algorithm to make intelligent suggestions, and we are spending some time looking through its results to see if it could become good enough to give to newcomers in a structured task. We are working in this Phabricator task.
Variants C and D engineering: this continues to be our main focus in terms of what we're building now, even as we plan for our next projects.
Measurement: this past week had the highest volume of newcomer task edits (1,103 edits) and editors (193 editors) since the feature's release. We believe this is caused by both the "guidance" elements of newcomer tasks, and the addition of Persian and Hebrew Wikipedias in recent weeks. The graph of numbers of editors by week is shown here. More details and graphs on our measurements can be found on the project page.Number of editors using newcomer tasks as of 2020-08-17
Hebrew Wikipedia: we have deployed the Growth features to Hebrew Wikipedia, which gets about 2,000 newcomers per month. Next, we're planning deployments to Russian and Polish Wikipedias.
Upcoming meetings on "add a link": although the team's engineering is currently focused on Variants C and D, we are planning for our work next quarter on our first structured task workflow of "add a link". Next week, the team will have a series of meetings about three main areas:
Data analysis: to help us predict how newcomers might engage with "add a link".
User experience design: to develop some design ideas that we'll share with community members.
Engineering: to make an architectural plan of how to store and serve link recommendations.
Persian Wikipedia: we have deployed the Growth features to Persian Wikipedia, which gets about 6,000 newcomers per month, making it the third largest wiki for Growth features. Next, we're planning deployments to Hebrew and Russian Wikipedia.
Language question: we have added a "language question" to the welcome survey in all Wikipedias that have Growth features. This question will help us find out how many newcomers may be capable of editing with the Content Translation tool. See the question in the accompanying screenshot.Welcome survey, showing the new language question.
Variants C and D: the team continues to work on these homepage variants, and will be engineering on it for the next several weeks.
The team has planned out how we will build Variants C and D, which are new version of the newcomer homepage that are meant to encourage more newcomers to try suggested edits.
We originally showed the designs in this update, but the images are included again here.
This will be the team's main project going forward in the coming weeks.
Other forthcoming work
In this task, we will soon be deploying a question to the welcome survey asking the languages that a newcomer knows.
In this task, we will soon be altering the help panel in Czech Wikipedia so that questions go to mentors, not the help desk. This will be a pilot to see if newcomers have better experiences through their mentors than the help desk.
Ever since newcomers tasks was deployed in November, we've known that the feature has "performance" issues. By "performance", we are referring to the time it takes for the feature to load on the page.
When newcomers visit the homepage, all the rest of the page appears immediately, and the newcomer tasks can take a couple seconds to appear. This is because queries and searches are running in the background to generate the lists of tasks.
This past week and next week, we are doing some work to measure exactly how long this step is taking for users around the world. We will use the data to decide if the experience is slow enough that the team should work on speeding it up.
Variant testing
We have been planning for the engineering work on Variant C and D, and we are starting the work this week.
Last week, we presented at the online-only Polish Wikiconference, showing the community how the Growth features work, and discussing whether they would be a fit for Polish Wikipedia. Slides are available here.
The work on guidance cleanup is now almost finished, and we're going to be moving on to future projects this week.
We continued to see elevated rates of newcomers clicking edit and saving edits on tasks, which we hope and think is attributable to guidance.
Language question
In the coming year, we are considering adding a new type of suggested edit to the newcomer tasks feed: an integration with the Content Translation tool. The idea is that, for newcomers who know more than one language, we would recommend articles to translate, or sections in articles to translate.
To prepare for this, we are adding a question to the welcome survey that asks newcomers which languages they know. This will help us have a sense of how many newcomers might be candidates for using the Content Translation tool, and it may also provide some interesting research results.
Update 2020-06-29: guidance cleanup and next projects
Last week was our second week of cleanup around the guidance release. We made many small interface improvements, as well as a few larger ones related to making it easier to navigate.
We may have a couple guidance things still to finish up this week, and then we'll be moving on to next projects.
Next projects
The main next project that the team will be moving onto is our next variant test: Variant C vs. D. This test will make suggested edits a much more prominent part of the newcomer homepage.
Before starting on that, we have a handful of interim tasks to do, including:
Piloting mentorship in the help panel, to see if newcomers and experienced users have a better experience by asking mentor questions in the help panel, instead of help desk questions.
We released guidance on June 15 in all wikis that have suggested edits.
We think we are seeing some early positive results from guidance.Number of distinct users saving a suggested edit, broken out by week and wiki. The week after the release of guidance had the most users since the suggested edits was first deployed.
Last week had the highest number of suggested edits completed since the feature was first deployed in 2019: 917 edits.
Last week also had the highest number of distinct users completing suggested edits: 139 users.
The team is now in the midst of spending two weeks cleaning up the feature, making small improvements, and fixing bugs. One change we're making will make it easier for users to navigate and minimize the help panel.
We spent last week testing, filing, and fixing bugs for guidance.
The biggest challenges have been around the logic and animations for how the panel displays the right information at the right time.
We'll be releasing to newcomers this week, and we'll continue to fix and adjust in the coming week or two. One example is this issue, in which we're going to make it easier to toggle between the guidance and the article that the user is editing.
Variant testing
After we finish working on guidance, the team will turn to our next variant test: Variant C vs. D.
This test will show suggested edits prominently on the newcomer homepage, and test different paths of onboarding the user to the feature.
The team is focused on finalizing the code for releasing guidance next week.
We have been testing, fixing bugs, and sorting out the edge cases around when and how the guidance should appear for newcomers.
We encourage everyone to start translating the many messages for the feature. To help, we've put together this translation guide, explaining where the messages fit into the feature.
Variant testing
In this update, we talked about results from Variant A vs. B. We found that Variant B, which has the user skip onboarding, leads to higher engagement with the suggested edits module.
Even though we had those results, we had not switched newcomers over to all receiving Variant B, because we also saw evidence that the onboarding may important for helping newcomers complete edits. Variant A seems to perform about 40% better than Variant B on mobile.
We now have enough data to look at whether one of the variants is more associated with completing edits. The result is that Variant A leads to more newcomers completing edits on moble, while the two variants do not have differences on desktop.
Therefore, we will be converting all newcomers over to Variant A while we do the work in the coming weeks to build Variant C and Variant D, our next test.
Overall, the test showed us that while Variant B's stronger affordance of the suggested edits module increases interaction, onboarding to the module is also important. We are using both these points in the designs for Variants C and D.Wireframe from our structured task explorations, showing how a user might leave feedback on an algorithm's suggestions.
We have continued to have good discussion about structured tasks. So far, about 30 users have participated across four languages.
This week, we talked a lot about how we might automatically suggest spelling fixes in arbitrary languages. You can see our notes here from a conversation with User:Beland, who created a system to search for spelling errors in English Wikipedia.
Our team's designer has created new wireframes exploring some of the concepts that we think would be important for structured tasks. You can see them all here, with an example posted here.
We are planning on releasing guidance during the week June 15.
This week and next week are about finalizing animations, instrumentation, and testing for bugs.
The written content for the feature is now available for translation, and we're encouraging all wikis using Growth features to translate it in the next week if they can.
We deployed the Growth features to French Wikipedia on 2020-05-19. So far, there have been 95 suggested edits from 36 different users. There have also been many mentorship and help desk questions.
Our discussion about structured tasks is continuing. We hope community members can visit the project page for structured tasks and give their thoughts.
So far, we've heard from about 20 community members across many different communities, and some of the main points of discussion are:
Generally positive feedback -- it sounds like we are thinking on the right track.
It is important that newcomers have the opportunity and are encouraged to learn to edit using the traditional editors, as opposed to only learning new structured editing tools.
Discussion of whether copyediting or reverting vandalism can be good structured tasks for newcomers.
Sketch of an idea for a structured workflow for adding links to an article, geared toward helping the newcomer edit at high volume.We hope community members can visit the project page for structured tasks and give their thoughts on the questions posted on the talk page.
This is a project we're in the middle of thinking about, and we want to hear advice and reactions from community members in as many projects as we can.
We plan to conduct this conversation over the next five weeks (through June 18). We'll be sending the information out in our newsletter and pinging users who we think would want to take part.
If you can translate the project page to your language, that will help us gather more community opinions.
We are continuing to finish up engineering work on guidance.
Regarding our previous update about having many translatable messages, we have decided to keep all the messages, and monitor whether communities struggle to translate them all.
Scaling
We are planning to deploy Growth features to French Wikipedia, which will be our first large Wikipedia. We'll start working on that deployment this week.
Update 2020-05-03: thinking about "structured tasks"
We are nearing the end of the engineering work for "guidance", currently working on the instrumentation.
One of our challenges is managing the many messages that will need to be translated. Since this feature guides users through completing suggested edits, it contains a substantial amount of text and examples that we wrote carefully to be helpful for newcomers. But since there are five different task types, with six or seven different steps, along with some examples sentences, and some differences between desktop and mobile -- there are about 50 messages to translate for this feature. This may be too many for communities to do easily and thoroughly. We are currently thinking about how to make this better, and welcome any ideas.
The idea is that we think we could break down editing workflows into a series of steps that newcomers can accomplish easily, especially with the assistance of machine learning.
This is an exciting idea, but there are also many potential pitfalls. This new project page is in progress, and after getting some more content on there, we'll start a community discussion so that we figure out the potential for this idea.
In our previous update, we mentioned that one idea for how we could more easily scale to bigger wikis is by converting the help panel over to ask mentor questions instead of help desk questions.
We're going to pilot this idea in Czech Wikipedia to see how it goes, and then decide if it is an improvement that can be brought to our other wikis.
Technical requirements are now written for our next test, Variant C vs. D. They can be found under this Phabricator tasks.
Scaling
As we prepare to deploy our features to larger Wikipedias, one important question is how we will scale the "help desk" and "mentorship features". Both these features require the involvement of experienced community members to answer questions. Large Wikipedias are expected to produce a large number of questions, and we want to make sure there are enough community members available to answer them.
One idea we have is to remove the "help desk" element from the help panel and to replace it with questions to mentors instead -- essentially creating one route for questions from newcomers: to their mentors. It will be easier for wikis to scale their mentorship capacity up and down than their help desk capacity. We think such a change could also enhance the newcomers' experience with suggested edits.
We want to try this on Czech Wikipedia in the coming weeks to see what the effects are on the volume of work given to mentors. The details are on this Phabricator task.
Last week, we deployed the newcomer tasks workflow to four additional Wikipedias that already had all the Growth features apart from newcomer tasks: Ukrainian, Hungarian, Armenian, and Basque.Mockup of Variant C on desktop
We also deployed all Growth features to Serbian Wikipedia.
We deployed all Growth features except newcomer tasks to French Wiktionary.
Together, newcomers on these new wikis have made 33 suggested edits since last week.
We finalized designs for our next variant test: Variant C vs Variant D. Variant C feature an initiated suggested edits module with optional onboarding. Variant D also features the newcomer tasks flow prominently, but immediately gives users the option to choose topics as part of mandatory onboarding. See mockups at right.
The analysis for our first variant test is now complete. It started in December, and split newcomers into these two groups:
Variant A: receive the original suggested edits workflow, in which they clicked a call-to-action to pass through two onboarding overlays before seeing the suggested edits module.
Variant B: arrive on the homepage with the suggested edits module being displayed, experiencing no onboarding overlays.
On desktop, Variant B yields double the interaction (clicking on anything in the suggested edits module), 60% more navigation (clicking on arrows to navigate to different suggested articles), and 30% more clicking on suggested articles.
On mobile, the variants perform the same for interaction and navigation, but Variant B leads to 15% less clicking on suggested articles.
Takeaways include:
Making the suggested edits module more prominent on the homepage gets more users to interact with it.
Though the overlays were not present on Variant B, they may still play an important role in giving the user context for what they're supposed to be doing with the module.
We will incorporate these takeaways as we design the next set of variants to test.
In terms of engineering, the team continues to work on "guidance", and it is possible to follow our progress in this Phabricator task.
Top-of-funnel work
We are now designing and planning around the project we'll be working on after releasing guidance: "top-of-funnel" work. We use this term because it is work that is meant to increase the number of newcomers who begin the "newcomer tasks conversion funnel". Right now, only about 20% of newcomers who visit their homepage interact with newcomer tasks, and we think that design changes can increase that.
We'll be doing this top of funnel work as a series of variant tests, which we started writing about here. Our very first variant test is almost complete, and the results will be posted next week.
You can look at some of the variant ideas we want to test in these mockups. They include ideas like:
Making the suggested edits module dominate the homepage.
Letting users peruse a list of task recommendations, instead of just looking at one option at a time.
Suggesting to newcomers who have previous edits that they edit articles they are reading.
The Growth team, and the rest of the WMF, are adjusting our pace of work as the world deals with the COVID-19 pandemic. We'll be working more slowly and more carefully in the coming weeks, allowing time for team members to take care of their families and themselves. See this blog post for a message from the WMF's CEO about how our staff is planning to adjust.
That said, our priority continues to be working on "guidance". The engineering work continues on that project.
We are also continuing to plan for how we will scale Growth features to many more wikis. We will begin reaching out to potential communities in the coming weeks.
The Growth team is currently planning our work for the coming fiscal year, which begins in July. We plan to build on the newcomer tasks project, which has been showing promising results.
In a previous update, we talked about evaluating a "link recommendation" algorithm. The idea is that a good algorithm could help us build a type of newcomer task that points newcomers to specific links that may need to be added -- giving them small tasks that fit their skill level, without requiring a huge amount of context.
We have since been developing this idea, and we're now sketching out an initiative we're calling "structured tasks". In short, we think that we can break difficult wiki editing workflows into steps that are much easier for newcomers to do, and much easier to do from mobile devices. We would put these structured tasks into the newcomer tasks workflow. They could include things like adding a link, adding an image, adding a reference, etc.
In the coming weeks, we'll post more information on wiki explaining this idea and asking for communities to react. While we think this could be a really exciting idea, it will be a lot of work, and there are many potential pitfalls. We'll need community thoughts to make sure we get it right.
We deployed ORES topic models to Arabic, Vietnamese, and Czech Wikipedias on 2020-03-05. Korean Wikipedian's models had some issues that are causing its deployment to be delayed.
In the coming weeks, we'll be publishing some information about how other Wikimedians and developers can access these models for their own work. As an example, one way to use them is with the "articletopic" search keyword. This works by typing something like "articletopic:sports" in a Wikipedia search bar to retrieve articles that are likely to be about sports. This page and links from it show the 64 possible topics to search.
This past week, we detailed exactly what user actions we'll log to understand how the feature is being used after we deploy it. The details are on this Phabricator task.
Update 2020-03-01: deploying ORES topics this week
We are planning to deploy our upgrade of the topic matching algorithm this week.
In our evaluation of the new ORES models, we found that the new models perform substantially better than the "morelike" algorithm we have been using for topic matching since Jan 13. We evaluated the models by looking at ten articles classified by the models into each topic, and then counting how many of the articles seemed to fit well in the topic.
The "morelike" algorithm averaged between 5.8 and 7.5 across the topics, depending on the language.
The ORES models averaged between 8.0 and 9.3 across the topics, depending on the language. The also have the added benefit of more topic granularity.
Engineering is underway to guide users through suggested edits via the help panel.
This past week, we drafted the language that will explain to users how to complete the different task types (copyedit, add links, add references, update article, and expand article). The challenge was how to give the right level of detail, while not overwhelming or confusing the newcomer. We want newcomers to complete a simple edit as quickly as possible.
We have also been deciding which data to record to understand the way that guidance will and will not be used.
We will soon be analyzing the results of our first homepage variant test, in which half of newcomers receive the suggested edits module already initiated on their homepage when they arrive.
As we await those results, we have designed additional homepage variants that attempt to combine the best of the variants from the first test. Those are being set up to be tested here.
We will also be designing and prioritizing a test that makes suggested edits the clear focus on the homepage, attempting to encourage more users to enter its workflow.
Update 2020-02-23: upcoming deployment of ORES topics
We are planning to deploy our upgrade of the topic matching algorithm either this coming week or the week after.
This upgrade will increase the number of topics available to newcomers from 27 to 64.
We will be altering the user interface to help newcomers navigate the larger number of topics. You can try out the upcoming interface in this prototype.
Planning to scale to more wikis
Over the past 1.5 years of the Growth team, we have deployed our features to nine wikis. As we learn that the features we're building are valuable to those wikis, we are now planning to deploy to many more wikis in the coming year.
The reason it takes work from staff and communities to deploy Growth features is that they have required community input to configure. For instance, community members supply our team with the most important links to help pages, and also supply us with a list of mentors for the mentorship module.
Our team is starting to make plans about how to streamline deployments so that we can work with more wikis more quickly. We'll be posting about those plans in coming weeks.
Engineering continues on loading ORES topic scores into the Search API, which will enable newcomer tasks to use them, and also enable other features to draw on topics.
Ambassadors are now evaluating the new scores to identify any topics that perform poorly. That evaluation is happening in this Phabricator task.
Link recommendation
All newcomer task suggestions are currently based on maintenance templates, a decision we made among many choices of where to source tasks to recommend.
Maintenance templates do not give newcomers much direction on how or where to edit in an article, and we believe they would benefit from having specific places in the article to edit.
We also see that many newcomers use newcomer tasks to add internal blue links to many articles.
Therefore, we are partnering with a researcher to attempt an algorithm that can recommend specific words or phrases in an article that should be made into blue links.
Ambassadors are evaluating that algorithm in this task, and we will decide within the coming weeks whether it might be good enough to incorporate into newcomer tasks.
Increased affordance of newcomer tasks
Coming up, we'll be working on some ideas to encourage more newcomers to try out the newcomer tasks workflow.
Homepage layout: we will test a variant of the homepage in which newcomer tasks is featured prominently as the main call-to-action.
Entry point while reading: for newcomers who have already done a newcomer task, we will create a design that suggests they edit articles they are reading that have maintenance templates on them.
Update 2020-02-10: beginning engineering on guidance
In watching user behavior after the deployment of topic matching on 2020-01-21, we see that about 75% of users who have the option to choose topics actually choose some. Though we'll need to take more time to see how topics have impacted the rate that users do suggested edits, this initial number is a good sign that users are interested in using topics.
The team is now primarily concentrating on upgrading the topics to use the ORES models instead of our initial "morelike" algorithm. This week, our team's ambassadors will be testing them to make sure they yield strong results.
We are also beginning engineering work on this part of the newcomer tasks workflow.
After completing user tests on the prototypes for this feature, we discovered a couple important findings:
Our mobile design, in which the panel "peeks" up from the bottom of the page, works.
Users are really interested in seeing demo videos of how to complete edits. Videos are difficult to produce, keep up-to-date, and scale, and so the team will need to talk in the future about how to address this need.
With that information from the user tests, we were able to create these versions of the prototypes, which are what we intend to build now:
We deployed topic matching to our four target wikis on 2020-01-21, meaning that all users with the homepage can access newcomer tasks and choose topics to narrow them.
We're now working on the next upgrade for topics: using the ORES models instead of our initial "morelike" algorithm. These models are expected to give better results, and we expect to begin using them in February.
We'll also be beginning the engineering work to add guidance to the newcomer tasks experience, so that newcomers understand how to edit the suggested articles once they arrive.
Deploying to additional wikis
On 2020-01-22, we deployed all the Growth features besides newcomer tasks to Ukrainian, Hungarian, and Armenian Wikipedias. These are our next three "non-target" wikis, meaning they receive all but our latest feature.
Our updated date for deploying topic matching is 2020-01-21.
This past week, we have been testing and fixing bugs.
Deploying to additional wikis
We are preparing to deploy Growth team features to Ukrainian, Hungarian, and Armenian Wikipedias.
These will be our next set of "non-target" wikis, meaning that they receiving features one step after our target wikis of Czech, Korean, Arabic, and Vietnamese Wikipedias.
Prototype of how the help panel will "peek" from the bottom of the page to offer guidance on mobileV1.1 (topic matching)
We are almost finished building the topic matching addition to newcomer tasks, using the "morelike" algorithm. This is planned to be deployed 2020-01-14.
After about a month, we will replace the morelike algorithm with the higher-performing ORES model. This model will be made available through the Search API so that other Mediawiki features will be able to draw on it.
After we deploy topic matching, the team will move on to engineering this final part of the newcomer tasks workflow.
We ran 12 user tests on desktop and mobile in December and we are currently analyzing the results to determine the final designs that we will build. These are the prototypes we tested:
We are still having active discussion on the discussion page for this project. We hope community members can join in. Some of the important thoughts being discussed include:
How the existence of Wikidata infoboxes might affect how we want to design this project, and where images should be placed in articles.
To what extent non-English speaking users can participate in this feature, given that most metadata in Commons is in English.
Next week, we'll summarize what we've learned so far from community conversations around this project.
Mockup of Variant C on mobile
Mockup of Variant C on desktop
Mockup of Variant D on desktop
_of_newcomer_homepage_2020-04-09.png)
_of_newcomer_homepage_2020-04-09.png)
