Talk pages project/Notifications
This page talks about the Editing Team's work to improve the notifications editors receive for the wikitext talk page conversations they are interested in.
|
Topic Subscriptions
Be made aware when someone is saying something relevant to you.
|
This initiative sits within the Talk pages project. This is the team's larger effort to help contributors work together more effectively. To accomplish this, we are building upon existing community conventions to evolve Wikitext talk pages. It is our intention to evolve talk pages in a way that gives experienced contributors more leverage to coordinate their work and connect with other editors, while making communicating on-wiki legible and intuitive for newer contributors.
To participate in and follow this project's development, we recommend adding this page to your watchlist. We will use this page to:
- Share and invite feedback on designs
- Announce deployment plans
- Share data about how the feature is being used
Status updates
editThis section contains updates about the project's development.
Receive notification when new topics are started
![Screenshot showing the new "Subscribe" functionality that [[phab:T263821|T263821]] introduced.](/wiki/File:Subscribe-Vector2022.png)
This week, people accessing talk pages using a desktop skin (e.g. Vector (2022), Vector (2010), Timeless, or MonoBook) who have the "Enable topic subscription" setting enabled in Special:Preferences will gain the ability to subscribe to entire talk pages.
Once "subscribed," people will be notified whenever a new topic is started on the talk page you have "subscribed" to.
See more in T263821.
Update to Special:TopicSubscriptions
When you visit Special:TopicSubscriptions you will notice two new columns within the table that contains all of the topics you have subscribed to:
- First subscribed: this new column shows you when you subscribed to a given topic
- Latest notification: this new column shows you the last time you received a new comment notification about a given topic
The Editing Team recently analyzed data from an A/B test of desktop Topic Subscriptions that ran at 20 Wikipedias from 2 June through 18 July 2022.
What follows are the conclusions we are drawing from this analysis and the next steps the team is taking as a result. You can find more details about these results in the Evaluate impact section below.
Conclusions
- Topic Subscriptions increase the likelihood that volunteers who have made < 500 cumulative edits will receive a response to the comments they post.
- There was a 8.6% increase in the percent of comments posted by Junior Contributors (people who have made under 100 edits) that received a response.
- There was a 15.3% increase in the percent of comments posted by people who have made 100-500 edits post that received a response.
- People who had Topic Subscriptions enabled respond to comments 57% faster (51-minute decrease in median response time) than people who did not have access to Topic Subscriptions.
The full A/B test report can be found here: Topic Subscription AB Test Analysis
Next steps
The Editing Team will offer Topic Subscriptions by default for all logged in volunteers, at all Wikimedia Wikis, beginning with the 20 Wikipedias that participated in the A/B test.
A/B Test
Yesterday, 2 June 2022, an A/B test of Topic Subscriptions began at 20 Wikipedias: Amharic, Bengali, Chinese, Dutch, Egyptian, French, Hebrew, Hindi, Indonesia, Italian, Japanese, Korean, Oromo, Persian, Polish, Portuguese, Spanish, Thai, Ukrainian, and Vietnamese.
This particular A/B test is intended to help us understand the impact Topic Subscriptions are having on the likelihood that contributors, across experience levels, will respond more quickly to comments that are posted on wikitext talk pages. Read more about the A/B test design.
Analysis
The team has completed an analysis of how people have been using Topic Subscriptions in the four months since the feature became available as an opt-in Beta Feature at all Wikimedia wikis.
In summary:
- People value having the option to subscribe to topics: 99.5% of people who received a new comment notification after manually subscribing to a discussion kept the feature enabled.
- People read the new comment notifications they receive: people read 96% of the new comment notifications they receive within two weeks of receiving them.
- No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.
The above is leading the team to be confident moving forward with plans to begin an A/B test to evaluate whether Topic Subscriptions cause people to receive quicker responses to the comments they post.
Note: you can see a more detailed report in the "Evaluating impact" section below.
Analysis
An analysis is currently underway to evaluate how people have been engaging with Topic Subscriptions while the functionality has been available as a beta feature (T280896). Specifically, this analysis will help us understand the extent to which people are finding Topic Subscriptions valuable and/or disruptive. This information will help us decide whether we think the impact of Topic Subscriptions is ready to be evaluated through an A/B test.
Usage
In October, we shared initial data about how people have been using Topic Subscriptions while the feature has been available as an opt-in beta feature.
Today, we have updated numbers to share:
- People have subscribed to ~17,400 distinct discussions
- People remain actively subscribed to 87% of the discussions they initially subscribed to
- Of all the discussions people are actively subscribed to, 18% of those subscriptions were initiated automatically and 82% of those subscriptions were initiated manually.
Automatic Topic Subscriptions
On 6 August 2021, we shared that we were working on functionality that would enable you to become automatically subscribed to talk page discussions you start or comment in using DiscussionTools.
Today, we are excited to share that Automatic Topic Subscriptions are available as a beta feature at most Wikimedia wikis.*
You can enable Automatic Topic Subscriptions by visiting Preferences and turn on Automatically subscribe to topics.
*For the list of wikis where Automatic Topic Subscriptions are not yet available, please review T290500.
Special:TopicSubscriptions
On 29 October, we shared that Special:TopicSubscriptions was now available. To make this page easier for people to discover, we have introduced a link to Special:TopicSubscriptions within Special:Preferences#mw-prefsection-echo.
Question: Are there other places within the interface where you would expect and/or value seeing links to Special:TopicSubscriptions? We are keen to hear the idea(s) you have in mind on the talk page or directly in Phabricator.
New Functionality
It is now possible to see all of the discussions you are currently subscribed to by visiting the new Special:TopicSubscriptions page on any wiki. We introduced this page, in large part, as a response to the feedback volunteers shared[1] [2] [3] [4] about seeking easier ways to:
- Find a conversation they had subscribed to and
- Unsubscribe from conversations they are no longer interested in
You can access this list of all of the discussions you are subscribed to by visiting Special:TopicSubscriptions on any Wikimedia wiki.
Scaling
.png)
Today, 19 October 2021, Manual Topic Subscriptions were made available as an opt-in beta feature at the English Wikipedia and all Wikimedia sister projects. This means, the feature is now available at all Wikimedia wikis as an opt-in Beta Feature.
Early usage
While we have not yet conducted a thorough analysis of how people are using the Manual Topic Subscriptions (this is being planned in phab:T280896) beta feature, we do have some early numbers to share. As of 13 October 2021:
- People have subscribed to ~4,000 distinct discussions.
- People remain actively subscribed to 80% of the discussions they initially subscribed to.
Offering Manual Topic Subscriptions at More Wikis
On 25 August, Manual Topic Subscriptions became available as an opt-in Beta Feature to all logged in users at all Wikipedias except at the English Wikipedia.
Offering Manual Topic Subscriptions at More Wikis
Today, 18 August, Manual Topic Subscriptions became available as an opt-in Beta Feature to all logged in users at Commons and Wikidata. You can see and subscribe to what volunteers at these projects have to say about the feature here: conversation on Commons and conversation at Wikidata.
Offering Manual Topic Subscriptions at More Wikis
In the coming weeks, we will begin offering Manual Topic Subscriptions as a beta feature at more projects. You can expect updates to be posted to this page once the list of wikis where the feature will be deployed next is finalized.
Notification Delivery
To increase the likelihood that Junior Contributors become aware when someone is talking to them, we will soon deploy a change that makes it so all new users will be notified about new comments posted in topics they are subscribed to by default. More information can be found in this task.
Automatic Topic Subscriptions
The first iteration of topic subscriptions enables you to manually subscribe to conversations by clicking or tapping the subscribe button that appears in each talk page's level 2 section headings.
The next iteration of topic subscriptions, which we are working on now, will enable you to automatically become subscribed to conversations after you do one of the following two things:
- Comment in a conversation
- Start a new conversation
Like other Talk Pages Project features, you will be able to decide for yourself whether you would like be automatically subscribed to topics, continue with subscribing manually, or turn the feature off entirely.
Today, 28 June 2021, Manual Topic Subscriptions became available as a Beta Feature to volunteers at the projects listed below. We will spend the next several weeks monitoring the feature to ensure it is working as people expect.
In the meantime, if you would like to see manual topic subscriptions offered as a beta feature at your project, please leave a message on the talk page.
Projects where manual topic subscriptions are now available:
- Wikipedias: Arabic, Czech, French, Hungarian, and Dutch
- Meta-Wiki
- MediaWiki
The team recently finished usability testing with Junior and Senior Contributors to identify areas where the Topic Subscription prototype did not meed their needs and expectations.
In short, Junior and Senior Contributor test participants found the Topic Subscription prototype to work as they intuitively thought it would/should. You can review the usability test findings in detail below: Usability testing.
These results are leading the Editing Team to feel confident offering Topic Subscriptions to a wider audience by deploying it as a beta feature at the projects listed below in the coming weeks.
- Wikipedias: Arabic, Czech, French, Hungarian, and Dutch
- Meta
- MediaWiki
Decisions
Over the past couple of weeks, we have made the following decisions about the first version of the topic subscriptions feature:
- The affordance for
Subscribing/Unsubscribingto/from topics will styled in the same way as the page's existing[ edit ]links and appear next to them. - To start,
Subscribing/Unsubscribingto/from topics andWatching/Unwatchingpages will be unrelated. The thinking for how we arrived at this initial decision can be found in the "Open questions" section of T279498's task description.
Looking ahead
- In the next couple of weeks, we will run a series of usability tests with Junior Contributors using usertesting.com to learn whether they can confidently and intuitively use the topic subscription prototype to become aware of new comments posted in conversations they are interested in.
Topic subscription prototype is ready
An in-progress prototype of the new topic subscription feature is ready for you to try.
This prototype enables you to elect to receive a notification via Echo when someone posts a new comment in any conversation you have decided to "subscribe" to.
To try the prototype, please follow the instructions on the talk page: Topic subscription prototype instructions.
Topic subscription prototype
Next week, the team will have a technical prototype of the new topic subscription feature ready for testing.
This prototype will enable you to elect to receive a notification via Echo when someone posts a new comment in any conversation you have decided to subscribe to.
We will post on the talk page when the prototype is ready for testing.
Special thanks to people like Enterprisey, GhostInTheMachine, Jack who built the house, Matěj Suchánek, Pelagic Sdkb, Timeshifter, and Vriullop whose feedback informed the decisions we made in building this initial version.
First intervention
The first intervention we will be working on to increase the likelihood Junior and Senior Contributors receive timely and relevant responses to the things they say will be a feature that enables people to subscribe to be notified about new comments in specific talk page topics.
The team has started the design and technical research that will be needed to offer this functionality. Next week, you can expect another update where we will share the requirements that will guide the first iteration of the topic subscriptions feature.
In the meantime, you can learn more about why we are prioritizing work on this particular intervention in the Background section below.
Note: we appreciate being able to more easily follow activity in specific talk page sections has been on peoples' minds for some time. If there are conversations, gadgets, research, etc. that you think would be valuable us to consider as part of this work, please boldly add it to the History section below.
Project start
The Editing Team is currently in the planning and technical exploration phase of this project. We plan to focus on the user-facing components of the implementation during the first three months of 2021.
In the time between now and then, you can follow the research we are doing for this project in Phabricator here: phab:T233447.
Objectives
editThis work is intended to increase the likelihood Junior and Senior Contributors receive timely and relevant responses to the things they say.
Where "the things they say" in this context means, the comments they post and conversations they start on wikitext talk pages, regardless of the tool used to publish these comments and conversations.
Once defined, you will be able to read about the key metrics the team will use to evaluate the extent to which the enhancements made as part of this project have been successful in achieving this impact.
Evaluating impact
editThis section contains the methods and data we will use to evaluate the impact of this feature.
Analysis 2: Impact
editIn the A/B test of Topic Subscriptions, we sought to answer three primary questions:
- Do Topic Subscriptions cause contributors, across experience levels, to respond more quickly to new topics and comments that are posted on wikitext talk pages?
- Do Topic Subscriptions cause contributors, across experience levels, to receive responses to a greater percentage of the comments they post and new topics they start?
- Do Topic Subscriptions create unwanted noise that discourages people from using talk pages?
To answer the questions above, we ran an A/B test of Topic Subscriptions on desktop at 20 Wikipedias. The results from this test can be found in the "Findings" section below.
Timing
This analysis was completed on 5 August 2022. The A/B test ran from 2 June through 18 July 2022.
Findings
- Topic Subscriptions increase the likelihood that volunteers who have made < 500 cumulative edits will receive a response to the comments they post.
- There was a 8.6% increase in the percent of comments posted by people who have made under 100 cumulative edits that received a response
- There was a 15.3% increase in the percent of comments posted by people who have made 100-500 edits that received a response
- People who had Topic Subscriptions enabled respond to comments 57% faster (51 minute decrease in median response times) than people who did not have access to Topic Subscriptions
- Topic Subscriptions reduce long lags in interaction
- Among people who had access to Topic Subscriptions during the A/B test, 0.4% of responses were published >10 days after the initial comment was posted
- Among people who did NOT have access to Topic Subscriptions during the A/B test, 7.3% of responses were published >10 days after the initial comment was posted
- Topic Subscriptions did not cause a sharp increase in the number of notifications people receive each day
- Prior to the A/B test, people at the participating Wikipedias received, on average, 3.6 notifications per day across
- During the A/B test, people at the participating Wikipedias received, received, on average, 4.2 notifications per day
- Topic Subscriptions did not cause fewer Senior Contributors to participate on talk pages.
Full Report
Topic Subscription AB Test Analysis
Analysis 1: Adoption
edit
In this first analysis, we sought to learn the extent to which Senior Contributors are trying Topic Subscriptions (Manual and Automatic) and finding them valuable.
This information is necessary for the team to decide whether the impact of the feature is ready to be analyzed through an A/B test that is being planned in T280897.
Meta
- Time Period Analyzed: 19 October 2021 – 1 March 2022
- Link to Full Report: Topic Subscriptions: Adoption Metrics Analysis (Github notebook)
Findings
Usage
- 28% of all people (40% of Senior Contributors) that edited a talk page and have the DiscussionTools beta feature enabled have subscribed to at least 1 topic since 19 October 2021 (when manual topic notifications was available as a beta feature to all Wikis)
- The majority (65%) of people actively subscribed to manual topic subscriptions are subscribed to 1 to 2 topics. 35% of manual topic subscribers have subscribed to more than 2 distinct topics.
Value
- People value having the option to subscribe to topics: 99.5% of people who who received a new comment notification after manually subscribing to a discussion kept the feature enabled.
- People read the new comment notifications they receive: people read 96% of the new comment notifications they receive within two weeks of receiving them.
- No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.
- Some people are using Topic Subscriptions a lot: while the majority of people (65%) who have manually subscribed to a topic have subscribed to 1 to 2 distinct topics, 6% of people are actively subscribed to more than 20 topics.
Disruption
- No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.
Background
editThe Editing Team is committed to evolving wikitext talk pages in a way that gives experienced contributors more leverage to coordinate their work and connect with other editors, while making the same communication and coordination practices and capabilities legible and intuitive for newer contributors.
A key part of these "communication and coordination practices" is people knowing when others are saying something relevant to you. Where "relevant" in this context could mean someone is talking to you directly or saying something in a conversation you have expressed an interest in.
The trouble is, people across experience levels find the existing notification system not as effective as they and we think it could be at making them aware when someone says something relevant to them.
Specifically, usability testing, the Talk Page Consultation, Community Wishlist Surveys (2015, 2021) and academic research has demonstrated people, across experience levels, find wikitext talk pages do not offer them a broad enough range of activity they can elect to be notified about. People also mention finding it difficult to understand how and where existing notifications are delivered.
"...pertinent changes to sections that interest me get superseded by subsequent changes to other sections."
"Sometimes the movement of the terrace is very large, and times when only one or two topics account for almost all editions on the terrace. When this happens editions in other topics may go unnoticed among others, then we don't even see that there are new comments, sometimes we don't even see that there is a new topic."—Machine translated from English Wikipedia and Portuguese Wikipedia discussion
In offering people a greater range of talk page activity they can subscribe to and improving the ways in which notifications about this activity are delivered to people, we are striving to increase the likelihood Junior and Senior Contributors know when someone is talking to them and for the person who is talking to them to receive timely and relevant responses to what they are saying.
Design
editThis section will contains information about the design strategy.
Deployment
editThis section will contain information about how and where the Notification enhancements are deployed.
Usability testing
editThis section contains information about the usability tests the team has run to identify ways in which the Topic Subscription feature could be improved to meet Junior and Senior Contributors' needs and expectations.
Test goals
The usability test we ran with Junior Contributors had two primary goals:
- Evaluate their ability to intuitively know what to do to receive notifications about new comments posted in conversations they were interested in
- Identify areas within the user experience where Junior Contributors' needs and expectations were not met
The usability test we ran with Senior Contributors had two primary goals:
- Verify Topic Subscription tool complimented Senior Contributors' existing workflows
- Identify areas within the user experience where Senior Contributors' needs and expectations were not met
Testing Conclusions
Both Junior and Senior Contributor test participants found the Topic Subscription prototype to work as they intuitively thought it would/should.
While test participants identified areas where the experience could be enhanced (more in "Detailed findings below"), nothing surfaced that caused the team to question the prototype in fundamental ways.
The majority of test participants were able to:
- Subscribe to a conversation
- Locate and understand the notifications they received when a new comment was posted in a section they had subscribed to
- Unsubscribe from a conversation they no longer wanted to be notified about
Testing Methods
The usability test we ran with Junior Contributors was run on user on usertesting.com (see test protocol) and included people who had used Wikipedia before and who have, at a minimum, edited once before. Test participants were asked to complete a set of pre-defined tasks with the prototype and to narrate what they thought and felt as they completed said tasks.
The usability test we ran with Senior Contributors was run on mediawiki.org, Arabic Wikipedia, and Czech Wikipedia. The test was open to anyone who saw the announcements made at these three wikis. Test participants were asked to experiment with the feature as they saw fit and to share their experiences in writing, on-wiki.
Detailed findings
Senior contributors noted the following potential areas to improve (T275232):
- Revise the appearance of the Subscribe affordance so topics are easier to distinguish: T279149.
- Introduce the ability to subscribe to all conversations on a talk page: T284795
- Introduce the ability for people to customize the kinds of edits they are notified about: T284794
- Make it easier to identify a new comment you have been notified about, on the talk page where said comment had been posted: T282029.
- Clarify the distinction between the two places – "Alerts" and "Notices" – where people can receive notifications (Junior Contributors noted this as well): T142981.
Junior Contributors noted the following potential areas to improve (T281438):
- Changing how comments are visually presented within conversations so they are easier to read, relate, and understand: T282269.
- Revise the appearance of the Subscribe affordance so it is easier to identify on the page: T282261.
- Introduce a way to view and manage all of the topics you are subscribed to: T273342.
History
editMany projects, conversations and research have, and are, working to improve contributors' experiences with talk pages. This project is better off for their existence. Some of the projects the team continues to learn from are listed on the main project page and below. If there is a project you think we should be aware of, please boldly add it here.
- User:Enterprisey's Section Watchlists
- pt:English Wikipedia and Portuguese Wikipedia discussion forums (1set2012)
- en:Wikipedia:Village pump (proposals)/Archive 93#Support or oppose a test Village Pump based on Portuguese VP
- Community Wishlist Survey 2015/Watchlists#Section watchlists
- Community Wishlist Survey 2016/Categories/Watchlists#CW2016-R043
- Convenient Discussions
- reply-link
- Flow satisfaction survey
- Moderated Testing, November 2014: talk pages and Flow
- Talk pages consultation
- Wikipedia:Global, cross-wiki, integrated or stacked watchlists. - Interwiki talk page notification.
Glossary
editThe Talk pages project glossary is intended to help us all communicate about talk pages more effectively by making sure we have a shared understanding about the words we use in our discussions and documentation throughout the project.