After posting initial numbers last week, our team's ambassadors went through 98 edits done through newcomer tasks, and found quite encouraging results:
90% of the edits are good faith
80% of the edits are productive
Many newcomers are correcting typos, adding links, and adding content.
We see that some users of newcomer tasks are doing their first or second edits. There are also many who created their accounts a few months ago, and started doing newcomer tasks after discovering the new feature on their homepage.
An improvement made in the past week increases the loading speed of the module.
After planning out the engineering work, we realized that the earliest that ORES models can be used for this feature is mid-February. Therefore, we have started building with an alternative algorithm: "morelike". We will deploy the first version with morelike and switch to ORES when ready.
The team has deployed several bug fixes and improvements to the initial version of newcomer tasks. One example is that all suggested edit cards now show the number of page views that the article has had in the previous two months. This is intended to convey to newcomers that their edit will have impact (see accompanying image).Screenshot of suggested edits module showing number of views
We have published initial numbers on how far users go into the newcomer tasks workflow, including the number of users who have made edits made so far (68 users).
We have started our first "variant test" for newcomer tasks, in which we test two versions of the feature against each other to see which performs better. In this first test, half of users will have the original workflow in which they go through overlays explaining the suggested edits module. The other half will arrive on their homepage with the suggested edits module already present. We will see which leads to more clicks and successful edits.
It is now possible to filter Recent Changes feeds to edits coming from newcomer tasks using the "#Newcomer tasks" tag. We hope that experienced users will use this to thank newcomers and encourage them in their work!
Technical work for making topic designations available for this feature work can be found here in Phabricator. The team is now moving most of our attention to this work.
Update 2019-12-08: newcomer tasks cleanup and first results
V1.0 (basic workflow)Mockup of concept for using the help panel to guide suggested edits
The team is continuing work on small bugs and UI improvements to this initial version of newcomer tasks. Though we're eager to move on to V1.1 (topic matching), we've decided that it's important to fix these issues so that they don't cause complications in the future.
The data we see for newcomer tasks so far is exciting. Healthy numbers of users are initiating the module and interacting with it, and we're seeing higher numbers of users than we expected who are clicking on tasks and editing them. In looking at some of the edits that newcomers make via the module, most look like they are in good faith and are productive. This is really encouraging news, because we expected few users to click on tasks without V1.1 (topic matching) and few users to complete edits without V1.2 (guidance).
We will publish some initial numbers within the next week. We'll also be asking our team's ambassadors to look at edits made through the module to get a sense of whether they are productive edits or not. This will inform our design for V1.2 (guidance).
The engineering work for this is beginning, with collaborations with the Search and Scoring teams to make topic model scores available for the Growth team's software.
We intend to release this version in January 2020.
V1.2 (guidance)
We are in the process of designing exactly how the help panel will be adapted to guide newcomers while they are doing suggested edits.
See the accompanying image for an example mockup of one of the concepts (the actual content is just a placeholder; i.e. we will not be including videos in this version).
On 2019-11-20, we released newcomer tasks in Arabic, Korean, Vietnamese, and Czech Wikipedias. All users who have the newcomer homepage will now see newcomer tasks, which means that half of newcomers will have a chance to see it, and the other half will remain in our control group, not receiving the homepage at all.
We were able to fix most of the important bugs before the release, but there is still clean-up work to do in the next week.
The deployment went well, and we see from the early data that newcomers are able to interact with all the parts of the workflow, with some going all the way to saving edits in suggested articles. We'll continue to watch this data to watch for issues and draw any possible early conclusions.
Over the next weeks, we'll be working simultaneously on the backend algorithm for ascertaining the topic of an article, and on the frontend interfaces in which users choose topics.
Update 2019-11-17: upcoming release of newcomer tasks
Mockup of how newcomers will select their topics of interestMentor training: our team's Czech ambassador held our first "mentor training" with four members of the Czech Wikipedia community in-person in Prague. Attendees said this training went well, and our next steps are to adapt it to be given online with communities that have members located far away from each other.
Newcomer tasksMockup of how newcomers will select their topics of interest
V1.0: we have spent the last week testing newcomer tasks and fixing bugs. The workflow is nearly ready to be released to our target wikis either this coming week or the following week.
V1.1 (topic matching): in addition to working with the Scoring team to improve topic matching methods, we'll also be working with the Search team to find a good way to store topic predictions of articles so that the newcomer tasks workflow can easily access them. We're talking with that team this coming week. We've also finalized our designs for this version, shown in the accompanying image.
V2.0 (additional task types): we chose to use maintenance templates as the source of tasks for the first version of newcomer tasks because all wikis have them, and there are many different kinds. They were the easiest way for us to get started with this feature. But they may not be the best sorts of tasks for newcomers to do. We're starting to work with the Research team on an algorithm that will recommend specific internal links for newcomers to add to articles -- as opposed to the suggestion to add links to an article "in general". We think this will be an even simpler task with which newcomers can get started.
Screenshot of newcomer tasks feature in Arabic Wikipedia
Mentor training: our team's Czech ambassador will be trying out a new mentor training in person in the Czech Republic. For those in the area, the signup page is here.
V1.0: we are now testing newcomer tasks v1.0 in both the beta and production environments to make sure that the designs are implemented correctly and that translations are strong. Please contact us if you would like to help test!
V1.1 (topic matching): we have started to work the Scoring team to improve the ORES model they have built to predict the topic of an article (Art, Music, Sports, etc.) The Scoring team will be working on clarifying the topic values used in the model and on making the model have full coverage in our target wikis.
The team is getting close to completing the engineering for V1.0 of newcomer tasks, which will move us into the testing phase.
Users who have their homepage turned on in Beta wiki can try out newcomer tasks. Please contact us if you would like to try it and need help. We're interested in any opinions or questions!
Planning meetings
We have been posting notes from our team's planning meetings last week.
Try out some suggested edits: we did a hands-on exercise in which we attempted to edit articles with maintenance templates. This helped us understand what challenges we can expect newcomers to face, and gave us ideas for addressing them.
Variant testing: after deploying the first version of newcomer tasks, we want to start testing different variants of the feature, so that we can improve it iteratively. In this session, we brainstormed different variants to test.
Update 2019-10-27: takeaways from planning meetings
The team continues engineering on the first version of the feature, to be released during November.
We are preparing for v1.1, an upcoming version that will add topic matching to newcomer tasks, so that newcomers can receive suggested edits based on their topics of interest. We have prototyped and evaluated three different methods for topic matching (see this Phabricator task for details). Early results look like a new ORES model for predicting the topic of an article may be a promising choice.
We are also preparing for v1.2, an upcoming version that will add guidance to the newcomer tasks workflow, so that newcomers who select articles to edit will have some explanation of what they're supposed to do. We'll use the help panel for this, and have some initial designs that can be seen in this mockup.
Planning meetings
Last week, the team held a series of remote planning meetings in which we spent concentrated time thinking about our long-term and short-term goals. Below are summaries of some of our sessions:A wireframe from the design exercise on creating a "dream homepage"
Integrated newcomer experience: the central question for the week was, "What does an integrated newcomer experience look like?" Our team has been building several different features for newcomers, but we believe they need to fit together into a cohesive experience. For instance, how might the newcomer tasks experience fit together with upcoming ideas for helping newcomers to build a user profile?
Try out some suggested edits: we wanted to make sure we understand what newcomers will experience when they use the upcoming suggested edits module (the module in the newcomer tasks flow). To do this, we each attempted to edit two articles that have maintenance templates, and took notes on what worked well and poorly.
Pre-mortem of newcomer tasks: we made a list of potential negative outcomes from newcomer tasks (e.g. "Very few newcomers click on articles to edit") and made reasons why those things might occur. We'll use this list to anticipate these outcomes and try to avoid them.
Newcomer tasks v1.1 (topic matching): after deploying the first version of newcomer tasks, we'll work on adding topic matching, so that newcomers can receive tasks that fit their interests. In this session, we reviewed the results of our different potential algorithms to determine which might work best for v1.1.
Newcomer tasks v1.2 (guidance): after deploying v1.1, we'll work on adding guidance for newcomer tasks through the help panel. In this session, we talked about initial designs and potential specifications.
Variant testing: after deploying the first version of newcomer tasks, we want to start testing different variants of the feature, so that we can improve it iteratively. In this session, we brainstormed different variants to test. For instance, we may test a version in which users proactively initiate the feature, against a version where the feature is already pre-initiated.
Sequencing our work: there is a long list of things the team wants to build and improve, and in this session we talked about the right order to work on those things.
Designing a dream homepage: in this design exercise, each team member developed an opinion on which homepage features could be removed, could be merged, moved to other places in the platform, or should be added. We each discussed how our "dream" homepage created an integrated newcomer experience.
The team is in the midst of engineering the first version of this feature, to be released during November.
We've completed a measurement plan that defines what questions we want to answer about the feature and what data we'll collect to do it. That will be posted in the coming weeks. We are currently working with other departments to make sure we are adhering to security and privacy best practices.
One innovation we plan is to be able to test multiple variants of newcomer tasks at the same time -- instead of just testing one variant against a control group. This will help us learn more quickly what works best for newcomers.
Planning meetings
Next week, the team will have a series of planning meetings, in which we'll work to more clearly define the future of our team and the newcomer experience we're building in the coming year.
We're looking forward to posting some of the things we accomplish during the week.
Update 2019-10-14: next set of designs for newcomer tasks
Newcomer tasksMockup of new "overlay" that explains newcomer tasks to mobile users
We completed user tests of the mobile version of newcomer tasks, and the notes are published here. Here are some highlights:
In general, users understood how to access and use the module, and what it was for.
They are surprised that “adding an image” is considered a difficult task.
The most important thing for us to improve is the guidance that users will get in the help panel while they are doing a task. It is challenging to show guidance on mobile while also showing the article to be worked on.
After the results from the mobile user tests, we've altered the designs a little bit and finished writing the wording for the feature. We have new versions of both the desktop and mobile designs, which the team's engineers are building now:
The team is finalizing our "measurement plan" for newcomer tasks, which lays out what data we'll record and how we will experiment with the feature to learn how to improve it. We'll be working on getting approvals from different WMF departments, and then publish a draft on wiki so communities can follow along.
Mentor trainings
Growth team features, such as the help panel and mentorship module, rely on experienced editors to answer questions from newcomers as "mentors".
In order to help experienced editors become better mentors, we have written a training and are planning to test it with the Czech community in the coming months. If it goes well, we will try to do the training with other interested communities.
The help panel was first deployed to newcomers in January 2019, and we have now finished analyzing data to determine its impact. A brief summary is below, and a more in-depth analysis is forthcoming.
In summary, the help panel has not shown an increase in activation (whether a user makes their first edit) or retention (whether a user returns to edit again).
The help panel does cause slight changes in edit volume (the number of edits made), but the results are conflicting. It increases edits during a newcomer's first day, but decreases them over the following two weeks.
Although increases in activation and retention are our team's ultimate goal, we also observe that large numbers of newcomers open the help panel (20%) and interact with it once open (50%). This tells us there is demand for the sort of help that the help panel offers.
Taken together, these results are disappointing -- we were certainly hoping that the help panel had a positive impact on the metrics. While we have many ideas for how to improve the help panel, we have decided to keep our attention on the newcomer homepage and newcomer tasks projects for the coming months. Those projects have promising early results and reach many more newcomers than the help panel does. The help panel will also be used as part of the newcomer tasks project, and will remain as part of the newcomer experience that the Growth team is building. We may revisit it in more depth in the future.
We have completed six user tests of the desktop version of newcomer tasks. All testers felt positively about the feature, and we have some notes on design changes to make. We'll post those next week.
Next, we'll be designing and testing the mobile version of newcomer tasks, as the team's engineers write the code for the desktop version.
We are also working on our measurement plan for the feature, deciding which interactions we'll record data for, and how we will A/B test it.
At the same time, we are prototyping approaches for "topic matching" -- how to show tasks to newcomers related to the topics they're interested in. This is a big challenge, and is tracked in this task.
Newcomer homepageShare of newcomers who discover their homepage increased when features were deployed at the two dotted lines.
In previous updates, we've mentioned our work on "features to aid discovery". Those are features that encourage newcomers to visit their homepage, hopefully ensuring that all newcomers at least know that their homepage is available. These features were deployed in July, and our goal was to double the number of newcomers who visit their homepage. We have now analyzed the results, and we see that we have exceeded this goal! See the graph at the right for details.
In Czech Wikipedia, the percentage of newcomers visiting their homepage has gone from 34% to 72%.
In Korean Wikipedia, the percentage of newcomers visiting their homepage has gone from 19% to 49%.
Since deploying the newcomer homepage in May 2019, we've looked at how often different components of the page are used. The page contains two ways for a newcomer to ask a question: the help module and the mentorship module. Users use the mentorship module to ask questions about 15 times more often than the help module. Therefore, to make the homepage simpler, are removing the ability to use the help module to ask a question.
The page includes these links to the video and slides from the Growth team's presentation about our work and learnings in the past year. Those links are a great way to learn more about our work, and we welcome any thoughts, reactions, or follow-up questions.
These are some of our topline notes:
Alignment on newcomer retention: It seems like Wikimania attendees generally believe that newcomer retention is an important problem.
Connecting offline to online: Many Wikimania attendees are very active in off-wiki events, such as edit-a-thons. Because of this perspective, we heard a lot of enthusiasm for ideas that connect our features better to offline events, such as making homepage mentors correspond to offline mentors.
Mentor dashboard: When talking about the mentorship module in the newcomer homepage, we heard experienced users request some sort of dashboard with which they could monitor newcomers who may need help, and have a way keep track of and thank them for their work.
The team is now engineering on the newcomer tasks project. You can follow along in the Phabricator tasks listed in this high level task.
Our evolving designs can always be found in these mockups (use arrow keys to navigate). Those mockups contain explorations of all the difference parts of the user journey, even though we'll only be building a subset of the features for the first version. See this list for what will be in the first version.
Next, we'll be running user tests on the mockups to further refine our design even as we engineer the code.
Update 2019-09-07: specifications almost ready for newcomer tasks
This week has been all about finalizing plans for the newcomer homepage.
Engineering work will start next week.
We've settled on plans for a first version that has these capabilities:
Modifications to the welcome survey to more clearly personalize the newcomer tasks
Adding a module on the newcomer homepage that provides suggested edit tasks.
Allowing users to filter tasks based on topics of interest and difficulty of task.
Giving users guidance through the help panel once they are attempting a task.
The two most difficult parts of this will be (a) filtering to topics of interest and (b) guidance through the help panel. One or both of these capabilities may not be part of the very first version, but will come soon thereafter.
Several members of the Growth team were at Wikimania in Stockholm.
We led sessions about our team's work and other growth-related topics.
We also had many interesting conversations with community members.
Full notes are forthcoming in a future update.
Mockup of possible design for newcomer tasks moduleNewcomer tasks
The team is focused on this new, ambitious project to recommend easy tasks for newcomers to get started with editing.
The project page now lays out our approach to sourcing tasks and to design. We plan to use maintenance templates to find tasks, and then to match them to newcomers' topics of interest.
We are in the midst of finalizing draft designs and investigating the technical approach for matching tasks to newcomers.
We will be moving on to engineering the backend and user testing the front end designs over the next week.
We hope community members can look over our plans and react on the talk page. If our plans don't look like they will work, we want to know right away!
The test of the welcome survey has completed in Arabic Wikipedia, and shows that the welcome survey does not negatively impact users and cause them to leave the wiki (the same result as on Czech and Korean Wikipedias). This means that we can safely use the welcome survey on that wiki to personalize newcomer tasks.
Wikimania starts this coming week, and several members of the Growth team will be present.
We are helping to organize the Community Growth space, which is a conference track focusing on how to bring newcomers into our movement and our projects.
This week, we have been bringing more technical definition to the project and we have been working on initial designs.
We are likely to use maintenance templates to find tasks to recommend to newcomers. All our target wikis use these, and they are the quickest approach to include in the initial version of the feature.
We are looking into algorithms that will allow us to match articles to newcomers based on the interests they describe in the welcome survey.
Since much of the team will be at Wikimania, we'll not make much progress over the next two weeks, but our priorities will be (a) technical evaluations of different approaches, and (b) designs for the "feature discovery" component.
Our team spent time planning and designing around the newcomer tasks project.
We've made some important decisions this week:
The newcomer persona we'll be thinking about most when designing is the Knowledge Sharer. This is a user who arrives on Wikipedia with some substantial edits in mind they want to make. We think we can increase their retention by giving them easy tasks that grow their skills, so that they are successful making the edits they arrived to make.
We're now thinking about the newcomer tasks project as being just one of the ways to help newcomers get started. Newcomers prefer to learn in different ways. Some may want to read a tutorial. Some may want to practice in the sandbox. Some may want to learn by doing. We want to give them choices, with newcomer tasks being the one we build first.
We decided that the top priority for the first version of task recommendations is the interface on the newcomer homepage where newcomers receive tasks, as opposed to prioritizing the quality of the tasks, or the guidance newcomers receive when doing the tasks. Those are important parts, but the top priority will be seeing how much demand there is for the tasks.
We've narrowed to a set of potential types of tasks to recommend, and currently thinking through them in terms of how easily they can be implemented and whether they will work well on our four target wikis of Czech, Korean, Arabic, and Vietnamese Wikipedias. This narrowed list includes things like using article improvement templates, the content translation tool, or adding links for orphan articles.
Other notes
Arabic newcomers have been using the help panel and newcomer homepage at levels comparable to the other three target wikis, with particularly high levels opening the help panel, and dozens of mentorship questions being asked from the homepage.
We are currently planning to develop a "mentor training" to help new communities learn how to mentor newcomers. We'll be taking notes in this Phabricator task, and welcome ideas and recommendations from everyone! The first training will be piloted in the Czech community.
This week, our team began a project to add task recommendations to the newcomer homepage. This will be a major project during the next six months.
We believe that recommending relevant and interesting tasks to newcomers will help them learn to make the edits they want to make, and give them a path toward contributing regularly.
Right now, the team is in a planning and design phase. We'll be adding more notes to the project page, but the big question right now is where to find the tasks to recommend to the newcomers. Please check out the page, and post any ideas or comments so far on its talk page.
Other notes
The newcomer homepage and help panel were deployed to Arabic Wikipedia this week. Arabic Wikipedia now has all the Growth team features.
The "discovery features" mentioned in previous updates were deployed to all our target Wikipedias this week, which is aimed at increasing the number of newcomers who find their homepage.
Feature to increase discovery of homepage from Special:ContributionsThis week, we'll attempt to deploy three features to increase the discovery rate of the homepage (they are listed in the update from 2019-07-12 below). See the accompanying image showing one of the features: a call to action on a newcomer's Special:Contributions page to encourage them to visit their homepage if they don't have any contributions yet.
We will also attempt to deploy all Growth features (welcome survey, help panel, and newcomer homepage) to Arabic Wikipedia, which is our fourth target wiki. The challenge with this deployment is that, unlike our other target wikis, Arabic Wikipedia uses Structured Discussions for its help desk page, and we needed to do additional work to make sure the help panel and homepage can post questions to that page correctly.
The team has set goals for the coming three months. These are part of longer-term objectives for the coming year that we will be discussing with communities in the weeks ahead. We'll be posting more information on all these goals and initiatives.
Newcomer homepage: increase activity through a task recommendations module. Now that we have seen several weeks of positive activity on the newcomer homepage, we think that the most important thing to add is a way for newcomers to find tasks to work on. The challenge will be recommending the right kind of tasks at the right point of their journey.
Newcomer homepage: increase feature discovery rate by 100%. Right now, only 20% - 30% of newcomers ever visit their homepage. We want to double that number by making sure all newcomers know how to find it via features linked from this task.
Deploy new experiments to four target wikis simultaneously (Korean, Czech, Vietnamese, Arabic). The team is now working with four target wikis, and we want all wikis to be receiving all our work at the same time, which increases our team's efficiency.
Deploy stable Growth features to five non-target wikis. Some of the Growth features, such as the welcome survey and help panel, are strong enough that wikis who want to try them are welcome to by working on this checklist.
Research and design for next project, either (a) structured user profile or (b) topic/neighborhood features. After we work on task recommendations, there are future developments to the homepage we want to plan. See our team's early notes on structured user profile here and on topic/neighborhood features here.
Help panel: increase usefulness through improvements to affordance, search, and UX flow. We have looked closely at data and anecdotes from the usage of the help panel, and we plan to pursue specific improvements to increase its effectiveness.
Mentor introductions: we've seen in homepage data so far that the mentorship module is much more popular for asking questions than the help module. We think that may be because of the "personal touch" that comes from having a specific mentor. To accent that personal touch, we have released the ability for mentors to customize their own introductory messages. See the screenshot at right and this set of guidelines for writing good introductions.
Renamed mobile menu item: to avoid user confusion between their homepage and the existing Wikipedia main page on mobile, we have altered the mobile side menu to refer to the main page as "Main page", instead of as "Home". See this task for images.
Priorities
We are currently prioritizing "features to aid discovery", in pursuit of our goal to increase the number of newcomers who discover their homepage.
GuidedTour after account creation: after creating their accounts, all newcomers will be informed of how to find their homepage through a GuidedTour.
Button on empty contributions page: many newcomers visit their Contributions page seeking to make their first contribution, but instead they find an empty page with no results. When their page is empty, a button will encourage them to visit their homepage.
Redirect from email confirmation: many newcomers confirm their email addresses, but the wiki page they land on after confirming doesn't explain what to do next. This feature will bring newcomers to their homepage when they confirm their email address.
Update 2019-07-05: mobile homepage release and homepage leading indicators
The mobile version of the homepage was released on June 27. Newcomers in our target wikis can now access their homepage both from the mobile devices and from desktop. We expect this to increase the number of newcomers who visit their homepage by about 50%.
As of July 1, the homepage is also activated in Vietnamese Wikipedia (both mobile and desktop).
Leading indicators
Before deploying the homepage, the team decided on a set of metrics that we would evaluate to see whether the feature is behaving in a healthy way. We've published our first evaluation of those numbers here. In summary, we are happy about the homepage's performance so far. We see about half of visitors clicking on something on the page, and many of them visiting multiple times. We also see that they are taking actions that the page recommends, such as confirming their email address, starting their user page, or asking a question to their mentor. One of the most interesting observations so far is that while 25 users have asked questions to their mentors, only one has asked a question to the help desk.
This week has been about fixing bugs and doing validation on the mobile newcomer homepage. We currently plan to release the mobile version during the week of June 24.
A couple of the most important issues have been:
Making sure that text in some of the longer languages (such as Czech) fit comfortable on a mobile screen. We developed "scrolling tabs" to help deal with this issue.
Allowing homepage modules to load as overlays instead of as new pages, which will make navigation faster and smoother.
Making sure that all our logging data is recorded correctly so we can evaluate the impact of the homepage.
The team is largely focused on the release of the mobile version of the newcomer homepage. We are continuously making small fixes as the software is being tested on the Beta Wiki.
About 30% of Czech and Korean Wikipedia accounts are created from mobile devices, so we expect this work to open up the homepage to a substantial number of newcomers.
Anyone can try out the mobile homepage by going to this link on a mobile device.
Growth features becoming available to more wikis
Several features built by the Growth team are now stable enough to be turned on for additional communities who want to experiment with them.
We have assembled this checklist that lays out the translations and configurations needed before the features can be turned on.
Please contact us if you are interested in moving toward turning on the features in your wiki.
The Growth team will begin to work with the Arabic Wikipedia community as a new target wiki. This is in addition to Korean, Czech, and Vietnamese Wikipedias.
Wikimania 2019 is coming up in August. The conference will include a "Community Growth" space, for sessions about how our communities expand through software and programs. If you have ideas for sessions, please submit them before the deadline on June 9.
After about a month of usage, we see a few interesting trends that make us believe usage is going well:
About half of users who visit the homepage click on a link or button.
About half of users visit the homepage more than once, with about a fifth of users visiting on multiple days.
Users are interacting with all the different modules on the page -- there is no clear favorite.
Users have been asking questions to their mentors -- but not on the help desk.
The primary work right now is about making the homepage work on mobile, which is expected to increase the number of newcomers seeing it. See the accompanying screenshot for the beta version of the software currently being tested on Beta Wiki. You can also see our notes from live user tests that we did of the mobile homepage here. We expect to deploy this to users in two weeks.
The team is also working on several features to help newcomers discover their homepage. Our goal is for all newcomers to learn that they have a homepage and how to find it.
Session about new editors at Wikimedia Hackathon 2019Members of the Growth team were at Wikimedia Hackathon in Prague from May 17 - 19. One of the three Hackathon focus areas was "new editor experiences", and this was an opportunity to help the participants understand the challenges of new editors so they could work on projects that helped.
Hackathon was also an opportunity to spend time with the members of the Czech community who have been answering incoming questions from newcomers via the help panel and newcomer homepage.
We held two sessions with participants, each of which was attended by about 20 people. We also held a special session for members of the Czech community.
We spoke to members of many different communities and started conversations about the potential for Growth team features to be deployed in their Wikipedias: Hungarian, Basque, Slovak, Ukrainian, Armenian, Tagalog.
Several Hackathon projects directly related to Growth team features or to newcomers in general:
We heard many interesting ideas from attendees as they learned about Growth team work:
We had multiple discussions about the benefits and challenges with live chat for the help panel. We talked about potential hybrid solutions that would solve the problems of volunteer coverage to answer chats, and challenges around moderation. We'll continue to have this conversation on our team.
Maybe after newcomers have their homepage for a while, their mentorship module could ask them if they want to become a mentor.
After creating an account, a user's experience barely changes at all. Some newcomers don't realize their account creation succeeded, and they create a second account.
Hackathon: many members of the Growth team will be at Wikimedia Hackathon next week, where we will have in-person time with the Czech community and will help participants pursue projects related to the Hackathon focus area of "new editor experiences". Please visit with us if you will be at Hackathon!
Newcomer homepageEmail confirmation screen -- a good moment to bring a newcomer to their homepage (shown in French Wikipedia)
After fixing a couple of small issues, we began the homepage experiment on May 6. Half of all new accounts in Czech and Korean Wikipedias now have access to their homepages on desktop by clicking their username in personal tools.
Because only 15 - 30% of newcomers will discover their homepage this way, we're working on two things to increase discovery.
First, we are working on the mobile version. This past week, we ran five user tests that showed we have a strong design, but that we need to be clearer in explaining the impact and mentorship modules. Full notes from these tests are forthcoming.
Members of WMF's Growth team working on designs for a potential "user profile" feature for MediawikiNewcomer homepage
The homepage was deployed in Czech and Korean Wikipedias on May 2, but is not yet being exposed to any newcomers. This weekend, community ambassadors in those Wikipedias will test things out to make sure there are not any major bugs.
The experiment will start on May 6, in which half of new accounts on those Wikipedias will have access to their homepages.
We've published our plan for how we will be tracking and measuring usage of the homepage here.
Vietnamese Wikipedia is still in the process of translating the feature and assembling a list of mentors. We will deploy in that wiki once those elements are ready.
The team is now starting to build the mobile version of the homepage, which is tracked here. The current mockups of the mobile experience can be viewed here.
We published the results of the first activation experiment, meant to detect whether the help panel causes more newcomers to make their first edit. This experiment finds no effect on activation from the help panel. This is disappointing, but it is only one of several ways we will judge the help panel. Other sources of data make us feel like the help panel has a good amount of potential that has not yet been unlocked. We will publish more information in the coming weeks.
The team is currently planning our next set of work for the help panel, which is meant to help more users find answers to their questions.
Week of planning activities
We published a set of notes from a team activity in which we thought about a "user profile" project to accompany the newcomer homepage.
Mockups of potential designs for mobile version of homepageNewcomer homepage
The homepage is planned for release on May 2 in Czech, Korean, and Vietnamese Wikipedias.
Once deployed, it will be available on desktop for half of all new accounts in those wikis as part of an A/B test.
After deployment, the next priority will be building a mobile version. See the accompanying mockups for how this design is starting to look, and please comment here with any reactions. We'll be user testing this design in the coming weeks as we begin building it.
Last week, the Growth team had several all-day planning sessions to talk about the future of the help panel, the newcomer homepage, and of the team's work in general.
We will be publishing notes and decisions from those sessions in the coming week.
Our prospective release date for the newcomer homepage has moved to May 2 because of technical deployment schedules.Mockup of a potential layout for the "Your recent questions" capability in the mentorship module.
The final touches of the homepage's UI are now complete and being tested.
We will be attempting to include one additional feature before release: an easy way for newcomers who ask questions to their mentors or to the help desk to navigate back to their responses. See the accompanying mockup, and please post any thoughts or ideas you have to the project's talk page.
Week of planning activities
Next week, the Growth team will be having a series of discussions in which we'll start to lay out the future of our existing features, and to plan for our next year. We'll discuss:
Evaluation of the help panel and options for continued iteration
Prioritizing the next set of features for the newcomer homepage
How our knowledge of newcomers and their needs has changed, and what that means for our longer-term strategy
We will report back on what we learned and discussed in two weeks.
Update 2019-04-05: planning for newcomer homepage release
We have set a prospective release date for the newcomer homepage of April 23. During that week, we plan to deploy the homepage to half of newcomers on Czech, Korean, and Vietnamese Wikipedias.
The user interface of the homepage is now falling into place, with final touches being worked on now. See a screenshot from Beta Wiki on the right (which looks a lot like last week's mockup!)
We are in the process of getting privacy and engineering approval for our instrumentation approach, and testing that approach.
We've finished designing the start module, and have begun to implement it. It's possible to click through its various states in these interactive mockups.Mockup of potential layout for newcomer homepage
We have also decided on a design and layout for the homepage as a whole. See the accompanying image, which shows that the start module has the highest affordance, and also highlights the help module on the righthand side.
We are now working on instrumentation of the homepage so that we'll be able to learn exactly how newcomers use it, and therefore be able to improve. That is under development in T219250.
We currently plan to deploy this feature during the month of April.
Before deploying the help panel, the team decided on a set of quantitative leading indicators that we would evaluate to see whether the feature is behaving in a healthy way. We've published our first evaluation of those numbers here. In summary, the help panel seems to be behaving in a healthy way, with good numbers of newcomers opening and using it. The leading indicators have exposed a couple areas for improvement, which we have worked on by introducing search and by displaying the feature in additional contexts.
Mockup of potential design for start moduleThe biggest takeaway from our live users tests of newcomer homepage mockups is that users are looking for a clear place to get started.
Learning from that takeaway, we reframed the "account completion" module to be a "start here" module that gives clear and concrete ways for newcomers to start learning or editing. It also gives newcomers feedback as they complete tasks, showing that they are done.
The latest mockup of this module is in the accompanying image. The detailed specifications are in this Phabricator task. We welcome any thoughts and reactions from community members on the project talk page.
Next week, we'll begin work on the instrumentation of the homepage so that we'll be able to understand how newcomers do and do not use it.
The experiment comparing versions of the welcome survey in Vietnamese Wikipedia has exposed that the abandonment rate for desktop accounts is substantially higher when those accounts receive the survey.
Curiously, mobile accounts do not have a higher abandonment rate when they receive the survey. The detailed statistics are in this Phabricator task.
We are not sure what is causing this difference in abandonment rate, but we turned off the survey in Vietnamese Wikipedia while we investigate. We want to make sure it is not causing good-faith newcomers to leave the site. It is possible that bot accounts are causing this.
Mockups of potential design for newcomer homepagePlans for the coming months
Our team has laid out our plans for the months of April, May, and June.
We have decided to focus our effort on iterating the help panel and newcomer homepage projects. We have decided not to continue work on the engagement emails project. We think that by focusing on two projects, we will be able to make more progress than by spreading across three.
We have decided what the initial version of the homepage will contain. See here for the details.
We completed five live user tests of newcomer homepage mockups and we are analyzing the results to decide whether our approach needs any changes. The accompanying image is our latest mockup.
We are also in the process of deciding exactly how we will instrument and measure usage of the homepage.
The latest code improves the visuals of the impact module and also adds the beginnings of the help module, which is like a help panel embedded right into the page. See the corresponding image f its current state on the beta wiki.
We are planning out how we will instrument and measure the homepage. It is more complex than our previous work, because a homepage with four modules is like a feature with four mini-features, each of which have different elements we will want to measure.
We have completed the experiment that compares Variation A to Variation C, which were two versions of the survey interface.
Variation A (the minimal interface) outperforms or ties with Variation C (the modern interface) on all measures, including response rate, abandonment rate, and activation rate.
Therefore, we are now showing Variation A to all new users in Czech and Korean Wikipedias, with no further experiments being run in those wikis. The experiment to compare surveys is continuing in Vietnamese Wikipedia.
Update 2019-03-01: first visuals for evolving newcomer homepage
Code has been merged that creates the homepage and adds the first module, the impact module. See the corresponding image of its current state on the beta wiki -- and note that we have not yet spent time on making the user interface look right.
Code is being written to increase "emailability" -- the number of newcomers who it is possible to be email. Changes will be tested to the "Special:CreateAccount" page and to the content of the verification email in our target wikis.
Update 2019-02-22: help panel expansion and newcomer homepage work
We have received some thoughts from community members on this project, and we're hoping for more. Please comment on the talk page!
We have decided to build the homepage as an additional "tab" in the User space, that newcomers will be redirected to when they click on their username.
We are considering ideas for how to increase the number of users that have verified email addresses. See this section to read about and give thoughts on the ideas we're working on.
It looks like Variation A (the simpler design) is getting higher response rates than Variation C (the more sophisticated design), in all wikis and platforms. We will publish more in-depth analysis, but it is likely that Variation A is the version of the survey that we continue to use going forward for all wikis that our team works with.
The early results from Vietnamese Wikipedia show higher abandonment rates than expected (people leaving the site after seeing the survey), and we're investigating those.
We began the "Personalized first day" project with the welcome survey so that we could gather information about what newcomers are trying to accomplish. The next step is to use that information to create experiences that help the newcomers accomplish their goal -- actually personalizing their first day. We asked for community thoughts in this update, and after discussing with community members and amongst our team, we are now planning two projects as next steps: "engagement emails" and "newcomer homepage".
To doEngagement emails: this project was first discussed positively by community members here back in September 2018, and the team how has bandwidth to pursue it. The idea is that newcomers who leave the wiki don't get encouraged to return to the wiki and edit. We can engage them through emails that send them the specific information they need to be successful -- such as contact from a mentor, the impact of their edits, or task recommendations. Please read over the project page, and comment on its discussion page with any ideas, questions, or concerns. Do you think this is a good idea? Where could we go wrong?
To doNewcomer homepage: we developed the idea for this project after analyzing the data from the welcome survey and EditorJourney datasets. We saw that many newcomers seem to be looking for a place to get started -- a place that collects their past work, options for future work, and ways to learn more. We can build this place, and it can connect to the engagement emails. The content of both could be guided by what newcomers say they need during their welcome survey, and contain things like contact from a mentor, impact of their edits, or task recommendations. Please read over the project page, and comment on its discussion page with any ideas, questions, or concerns. Do you think this is a good idea? Where could we go wrong?
Existing projects
Help panel: about 20% of users who see the help panel open it up. About half of them click on a help link. And about 5% of Czech users and 1% of Korean users ask questions. 20 questions have been asked in Czech Wikipedia, 4 have been asked in Korean Wikipedia. Since many people are trying to find help themselves, and fewer are asking questions, we're going to deploy the ability to search for help, likely next week. We've also published our written plan for measurement and experimentation on the help panel here.
Welcome survey: comparison of Variation A and C continues, in order to determine which gets a higher response rate.
Update 2019-01-24: welcome survey in Vietnamese Wikipedia
Our A/B test of the welcome survey has been analyzed, which shows that receiving Variation A of the survey does not significantly decrease the rate at which newcomers edit.
The welcome survey has been deployed to Vietnamese Wikipedia, with half of newcomers getting Variation A and half getting Variation C.
For the help panel, 9 questions have been asked in Czech Wikipedia, and 0 have been asked in Korean Wikipedia, after about 200 users in each wiki have seen the help panel button, with about 40 users in each wiki opening it up.
We are developing an additional feature for help panel: the ability to search for help content. This feature was appealing to user testers, and would search in the Help and Wikipedia namespaces.
We are continuing to plan upcoming projects, with project pages to be posted in the coming days.
Update 2019-01-18: initial report on EditorJourney
This initial report on what new editors do on their first day is now published on wiki. Please feel free to add comments on questions to the talk page, or to translate it into your language.
Variation C of the welcome survey is now being tested against Variation A in Czech and Korean Wikipedias, to determine which one gets higher response rates.
The help panel has now been deployed for a week, and we are seeing substantial numbers of newcomers opening it and clicking links. Asking questions has been more rare: 4 questions have been asked in Czech, and none have been asked in Korean.
We are beginning to work with a third community: Vietnamese Wikipedia. The EditorJourney schema is now collecting data, and we will be starting the welcome survey there next week.
We are planning new projects to use the data from welcome surveys. We will post initial wiki pages about those projects next week.
Update 2019-01-11: help panel deployed in Czech and Korean Wikipedias
Yesterday, we deployed the help panel to Czech and Korean Wikipedias. This is the Growth team's first feature built specifically to increase new user activation (editing for their first time) and retention (coming back to edit again). Going forward, half of new accounts will see the help panel button in the lower right of their screen when they open any editor on any page -- both on desktop and mobile (but not mobile apps). The other half of users are in the control group of our experiments.
We carefully instrumented the feature so we would be able to tell exactly how users do and don't use it, and we're going to be watching the data closely to see how many people open the panel, click on links, and ask questions. After about 24 hours of the feature being live, about 25 users have seen the help panel, with only one question having been asked so far -- although several users have opened the panel and clicked on links inside. We'll post more detailed numbers next week. To follow along with the help panel activity, feel free to check out these links:
The welcome survey was deployed such that half of newcomers would receive it, with the other half in a control group. This was so that we could detect whether receiving the survey caused newcomers not to edit -- the concern was that putting up another barrier to editing would deter them. We are happy to say that the experiment shows that the survey does not inhibit editing, and we will publish these results in the coming weeks. Because of this, we feel comfortable testing another version of the survey: Variation C. We're going to be showing Variation C to half of newcomers, and the original Variation A to the other half. We want to find out whether Variation C's more dynamic and streamlined interface increases participation. We'll know the initial results in about four weeks.
Update 2019-01-03: initial results from welcome survey
Right now, the team is mostly focused on our upcoming launch of the help panel next week, executing on the plans described in previous updates to test and launch on time. But the most exciting new thing we are looking at and thinking about is the initial report on the welcome survey results.
The welcome survey launched in Czech and Korean Wikipedias on November 19th, and we collected hundreds of responses in each wiki in the following month. Our team had a goal of generating an initial report by the end of December.
The initial report is here, in which we can see the reasons people create accounts, the topic they're interested in editing, the Wikipedia experience they have already had, and their interest in getting help from experienced editors. Here are some toplines, with the rest in the report:
The survey has high response rates: 67% in Czech and 62% in Korean.
Many people create accounts just to read articles -- a potential opportunity to engage new editors.
High numbers of people indicate they are interested in being contacted for help: 36% in Czech and 53% in Korean.
We're still digesting and thinking about these responses, and discussing how to take action on them. We're pleased to see that many people respond to the survey and take it seriously. We're seeing a pretty wide spread of intentions and topics people are interested -- this makes us optimistic that we can successfully personalize the first day.
A high priority analysis that we're working on now is figuring out whether being shown the survey makes a newcomer leave the wiki, instead of continuing on to edit. This is the main concern raised by community members around this idea, and it's the reason we have only been displaying the survey to half of respondents, keeping the other half as a control group. Early results seem like the survey is not decreasing activation (our word for "making a first edit"), and we will publish those results in more depth.
The help panel is still on track to launch next week to Czech and Korean newcomers.
The text has been largely translated on Translate Wiki, and the team is currently testing and fixing bugs.
We measuring using the new HelpPanel EventLogging schema, and we received both engineering and privacy approvals on our plan. We'll be publishing that plan on wiki in the coming weeks.
To do Engagement emails: this project was first discussed positively by community members here back in September 2018, and the team how has bandwidth to pursue it. The idea is that newcomers who leave the wiki don't get encouraged to return to the wiki and edit. We can engage them through emails that send them the specific information they need to be successful -- such as contact from a mentor, the impact of their edits, or task recommendations. Please read over the project page, and comment on its discussion page with any ideas, questions, or concerns. Do you think this is a good idea? Where could we go wrong?