Growth/Personalized first day/Variant testing

Starting in December 2019, the Growth team started testing variants of features against each other -- instead of just building one version of a feature and seeing whether it does or does not increase new editor retention. Testing variants will help us learn and iterate faster. This page contains running lists of variants we want to test for our different features, which we'll draw from when planning tests going forward. The lists are long and exhaustive, but we expect to test under 10 of these per year.

Newcomer tasks variantsEdit

Below is the list of feature variants we would be interested in testing for newcomer tasks. Competed tests have summarized results in the table and full results in the section below.

Name of test Component Variant A Variant B Question/Result Status
Initiation Start module Users click a button in the start module to initiate the suggested edits module, and proceed through overlays Suggested edits module is already present on the homepage the first time a user visits. Variant B yields more interaction, navigation, and clicking, but only on desktop. On mobile, the variants perform the same, with Variant B leading to less clicking. Subsquent analysis showed that Variant A led to more users editing on mobile. Deployed 2019-12-13 in cswiki, viwiki, arwiki, kowiki. Analyzed 2020-04-06. Switched all newcomers to Variant A on 2020-06-08.
Homepage takeover Suggested edits models Pre-initiated module has clear call-to-action that opens overlays Pre-initiated module has two buttons that each open up an overlay How can a pre-initiated module also give the user exposure to the valuable overlay information? Engineering now
Task list Suggested edits module Suggested edits module shows one suggestion at a time Suggested edits module lists many suggestions at once Is it better for the user to be focused on one task at a time? Or see each suggestion in context of the others?
Mobile initiation Mobile Suggested edits is initiated from inside the start module Suggested edits can be initiated via its own card on the mobile homepage Is it confusing to split suggested edits out of the start module to give it higher affordance? Or does that make more newcomers initiate it? Engineering now
Start module Start module Have only one start module, with several ways to get started Split the start module into "profile" and "editing" start modules Will it make more sense to newcomers to further organize the start module? Or will it make it confusing where to start?
Affordance Suggested edits module Normal sized homepage module Suggested edits module takes up more of the screen as an overlay or separate page Will giving the module higher affordance allow the user to focus on it? Or will it distract them from their homepage? Engineering now
First suggestion Suggested edits module First suggestion is copy edit First suggestion is add links Do different task types have different levels of attractiveness?
Personalization All Language is personalized to newcomer's stated goal throughout (e.g. "add a photo") All newcomers get generic messaging Does our personalization of language increase or decrease engagement?
User profile Start module Suggested edits is initiated from the start module Suggested edits is pre-initiated, and the start module instead contains the original user profile call to action. When users are given the option to either do suggested edits or work on their user profile, which do they choose and in which order?
Edit mode Guidance Newcomers arrive on an article in read mode Newcomers arrive on an article in edit mode Will newcomers learn to click "edit" for themselves if they arrive in read mode? Or should we get them right into the editing context?
Help media Guidance Newcomers read text about how to complete their tasks Newcomers see videos or animations about how to complete their tasks Are non-textual media a better way for newcomers to learn how to edit?
GuidedTour for module Suggested edits module Newcomers learn the suggested edits module on their own A GuidedTour points out different parts of the suggested edits module Can newcomers figure out how the suggested edits module works just from its design? Or do they benefit from having it pointed out to them?
GuidedTour for guidance Guidance No GuidedTour A GuidedTour can be initiated from the help panel to point out things like a maintenance template or the "link" button Can newcomers understand how to edit just from the text of the help panel, or would they benefit from pointing things out?
Suggested edits everywhere All Normal newcomer tasks workflow The help panel contains suggested edits, and is openable from anywhere in the newcomer's experience, on any page. It could open if the newcomer is trying to edit an article without a maintenance template, to suggest a different article. Should we centralize all newcomer tasks in the homepage? Or can users start them from just the smaller help panel?
Call to action Suggested edits module Newcomer must click on article to start editing it A clear call to action like "Start editing" is in the suggested edits module Will newcomers know they are supposed to click an article to edit it? Will an encouraging call to action promote that behavior?
Initiation button Start module Call to action says "See suggested edits" Call to action says "Try an edit!" What text will be most honest and persuasive to get newcomers to try suggested edits?
Mentor notification All Normal newcomer tasks workflow When a newcomer makes a suggested edit, notify their mentor automatically, encouraging the mentor to check the edit and/or thank the user. Can we productively add mentors to the newcomer tasks process?

General homepage variantsEdit

Below is the list of feature variants we would be interested in testing for the newcomer homepage.

Name of test Component Variant A Variant B Goal
Avatars Mentorship module No images or avatars of mentors Mentors can add an image or photo to represent themselves Will increasing the "personal touch" of mentorship improve and increase interactions with mentors?
Customize homepage Homepage Normal homepage Allow newcomers to remove or move modules from the homepage Will newcomers use their homepage more if they can customize it to only contain the modules they want?
GuidedTour for homepage Homepage Newcomer figures out how to use the homepage on their own Newcomer gets a GuidedTour pointing out the different homepage modules Will newcomers understand the homepage based on its design, or do they need guidance?
Mentor calls to action Mentorship module Normal mentorship module Offer other ways for a newcomer to contact their mentor, as opposed to just "Ask a question", such as "Introduce yourself" or "Ask for a review of your edits" Will newcomers engage better with mentors if they have specific calls to action?
Hi to mentor Mentorship module Normal mentorship module Send notification to user encouraging them to say hi to their mentor. Will encouraging this connection cause it to occur and endure?

ResultsEdit

Initiation (A vs. B)Edit

  • On desktop, Variant B yields double the interaction (clicking on anything in the suggested edits module), 60% more navigation (clicking on arrows to navigate to different suggested articles), and 30% more clicking on suggested articles.
  • On mobile, the variants perform the same for interaction and navigation, but Variant B leads to 15% less clicking on suggested articles.
  • On mobile, Variant A leads to 40% more newcomers editing than Variant B, but the edit rates are statistically the same on desktop.
  • Takeaways:
    • Making the suggested edits module more prominent on the homepage gets more users to interact with it.
    • Neither Variant A or B successfully made the module prominent on mobile.
    • Though the overlays were not present on Variant B, they may still play an important role in giving the user context for what they're supposed to be doing with the module. It's possible that this is reflected in the data in two ways:
      • Variant A yields 40% more newcomers editing on mobile than Variant B.
      • The increases between Variant A and B on desktop is lower with each successively "more engaged" type of engagement. In other words, simply interacting with the module is increased by 100%, but navigating between articles is only increased by 60%, and selecting an article is only increased by 30%. This may show that users don't have sufficient context to understand that they are supposed to navigate between and select articles.
  • Next steps: we turned Variant A on for all newcomers while we build Variants C and D. This is because it is shown to yield more newcomers completing edits.
  • We'll use these learnings to design Variants C and D, with work happening here.