Growth/Community Updates

Group:	Growth
Start:	2024-02-23
Backlog:	Growth backlog
Lead:	Tech Lead: Sergio Gimeno Product Manager: Kirsten Stoller
Updates:	Most updates posted here.

Translate this page

Tracked in Phabricator
Task T360485

This page describes the Growth team’s work adding a new module to the Newcomer homepage. This new module can be used by communities to share important community news, campaigns, events, or WikiProjects. This module will be Community Configurable to allow for easy customization.

How to use the Community Updates module.

Current Status

23 February 2024: Initial project page created
28 June 2024: Initial version testable on Beta wikis
October 2024: Initial version testable at Pilot Wikipedias (Arabic, Spanish, French)
December 2024: Publish experiment results
Up next: Release the Community Updates module to all Wikipedias

Summary

The Growth team will create a new module on the Newcomer homepage that can be used by communities to highlight important community events and initiatives. This Community Updates module will be Community Configurable, so communities can decide to highlight specific events, projects, campaigns, and initiatives, that are set by admins using Community Configuration.

As newer editors progress in their involvement on the wikis, the Newcomer homepage should evolve and offer more complex tasks and opportunities that encourage new editors to engage in more meaningful ways. By exposing newer editors to important community events and initiatives, newer editors will hopefully get more invested and involved in the wikis and connect with other Wikimedians.

Furthermore, this new module can be used to help bring attention to content gaps on high-impact topics, which aligns with the 2030 Movement strategy’s “Topics for Impact” recommendation.

Hypothesis

If we implement a Newcomer homepage module with community configurable text, icon, call to action, and target audience settings, then the banner can be used to highlight knowledge gaps, and more editors will improve articles related to high-impact topics.

To evaluate this hypothesis, we'll examine several key questions and track corresponding metrics:

Do Communities utilize this new module?
- Metric: The number of wikis that utilize the new Community Updates module within 2 months after the launch.
Is this module relevant to Newcomer homepage visitors?
- Metric: The percentage of Newcomer homepage visitors who click through to learn more.
Does this module help communities highlight important events, campaigns, and projects?
- Metric: The number of Newcomer homepage visitors who sign up for the tested community event/campaign/project compared to the control group (who do not see the new module).
Does this module help fill knowledge gaps?
- Metric to be measured separate from this experiment: An increase in the portion of newly created or improved articles on high-impact topics with acceptable quality, per the “global quality score”, that are created or edited on Wikipedia, starting with underrepresented geographic regions and gender. (This is the 2023-2024 WikiExperiences 1.3 Key Result from the WMF Product & Technology department's Objectives and Key Results.)

Target Audience

This project will primarily target Learners: registered editors whose experience falls between Newcomers (fewer than 10 edits or 4 days of activity) and Experienced Editors (500+ edits, >30 days of activity).

We will target both mobile and desktop Newcomer homepage visitors.

Our primary wiki audience is mid-sized Wikipedias, but the feature will be Community Configurable so that it can be adjusted to work well for all Wikipedia language editions.

Research

The project is guided by many previous research projects and experiments:

Community Discussion

The Growth team will initiate a discussion on our current Pilot wikis (Arabic Wikipedia and Spanish Wikipedia), along with a conversation in English on the associated MediaWiki talk page. Questions for Community Discussion:

Do you have any feedback or concerns about this project idea?
Do you have any suggestions for making this feature more impactful or meaningful to your community?
This feature will be designed to be Community Configurable by Admins. Do you think this feature will be used on your wiki?

Design

The following are initial design mockups of the mobile layout of the Newcomer homepage, including two new modules: the Community Updates module and the Content Translation task.

English mockup
Spanish mockup
Arabic mockup

The Community Updates module will be Community Configurable. Admins will be able to enable or disable the new module and decide how experienced an editor should be in order to see this module (based on edit count). The content of this module is entirely configurable as well, including the module's title, image, text, and call to action.

Community Configuration for the Community Updates module

Measurement and Results

What we will measure as part of the A/B test (T365889):

Question	Metric	Event to be tracked
Awareness - How many account holders visited their Homepage while this feature was visible?	Number of users who visited the Homepage during the experiment	Homepage pageview
Consideration - What is the click-through-rate (CTR) of the module?	Number of impressions of the Community Updates module	Module impression
	Click Through Rate (CTR) of Community Updates module	Module click
Conversion - Do more newcomers sign up for the WikiProject or Campaign when they are exposed to the Community Updates Call To Action (CTA) on the Homepage?	Number of users who signed up for a campaign during the experiment	Campaign signup

Instrumentation: No new schemas will be created for this as we’re using the Metrics Platform Instrumentation and will be relying on the core schema. We will use the HomepageVisit schema to track pageviews of the Newcomer Homepage. We also plan to mirror the Community Updates module instrumentation using the existing HomepageModule schema, as that enables us to learn about possible differences between the two approaches.

Data retention plan: We do not plan to retain any data for longer than the standard 90 day data retention policy.

Experiment Results

The Community Updates module experiment ran on four wikis: the Arabic, Czech, French, and Spanish Wikipedias. For the purposes of analysis, we consider 2024-11-01 (at midnight) as the deployment time and examine activity on those wikis during the month of November. During that time, two of the wikis ran updates that had limitations on who could see the update: the Arabic and French Wikipedia required at least 10 edits. On Czech and Spanish Wikipedia there was no limitation.

Another limiting factor in understanding the effect of the module is that the purpose of the updates were different from wiki to wiki. For example, on Spanish Wikipedia that Community Updates module linked to Wikipedia Asia Month 2024, while on Czech Wikipedia it linked to information about their weekly online wiki advice sessions.

The analysis was limited to the Spanish Wikipedia because of the time available to perform this analysis, the wiki's size, that there was no limit who could see the module, and that the module linked to a campaign that required users to sign up (meaning that we could track who signed up).

During the experiment period, the module was shown to about 3,500 users who created Wikipedia accounts during the experiment, and about 1,000 users who had registered previously. We limited our analysis to the former group as the Growth team focuses on the newcomer experience, and also because the impression and click-through counts for that group were substantially higher.

The per-user click-through rate of the module during the campaign was 1.7%. There was very little difference between the desktop and mobile web platforms when it comes to this rate. This rate is low but similar to baselines we gathered prior to the experiment for the "banner module", a previous feature on the Newcomer Homepage that worked similarly to the Community Updates module.

The sign-up rate for users after they clicked on the link in the Community Updates module, less than 5%, was substantially lower than baselines from other experiments. This led to an overall Campaign Sign-Up Rate, the key metric of the experiment that was also substantially lower than expected. When combined with the fact that the sign-up rate in the control group was also small but not zero, we end up with overall results that are not statistically significant.

Ideas for further improvements

The Growth team aims to release experiments quickly, so the first version of this feature may have more limited functionality. After initial experimentation, if this feature proves valuable, then we should consider adding many further improvements which are detailed in: Community updates module - second iteration improvement ideas.

Deployment

Although the initial experiment did not produce statistically significant results, the data suggests that newcomers who saw the Community Updates module may have been more likely to sign up for a campaign. Additionally, we’ve received encouraging feedback from communities and affiliates, who view the module as a valuable tool for sharing local initiatives and onboarding newcomers. In response to this feedback, we will make the Community Updates module available to all Wikipedias in a disabled state. This enables local administrators to decide how and whether to use the feature, as it can be fully configured by communities via: Special:CommunityConfiguration/CommunityUpdates.