Talk pages project/Replying
This page talks about the Editing Team's work to improve contributors' workflows for replying to comments on talk pages, across Wikipedia's 16 talk namespaces. This new workflow for replying to specific comments is intended to make participating productively on talk pages easier and more intuitive for Junior and Senior Contributors.
|
Talk pages
Help contributors respond to comments on talk pages more easily.
|
This initiative sits within our team's larger effort to help contributors work together more effectively. To accomplish this, we will build upon existing community conventions to evolve talk pages. It is our intention to evolve talk pages in a way that gives experienced contributors more leverage to coordinate their work and connect with other editors, while making communicating on-wiki more accessible and intuitive for newer contributors.
To participate in and follow this project's development, we recommend adding this page to your watchlist. We will use this page to do things like:
- Share potential designs and ask for feedback about them
- Test prototypes
- Post updates about this feature's development
- Ask and invite questions about any part of this project (e.g., measurement, deployments, etc.)
What it is
The reply tool is an extra button that appears at the end of a post on a talk page. When you click on it, it opens a reply form that makes replying to that post easier to do correctly. It indents correctly, helps ping correctly, and automatically signs correctly for you, among other things.
See Help:DiscussionTools and Talk pages project/Feature summary (screenshots!).
Technical information
The replying feature is implemented via the DiscussionTools extension.
Troubleshooting information is available at Help:DiscussionTools/Why can't I reply to this comment?
Status updates
This section contains updates about the project's development.
The scale of the Reply Tool's usage has grown beyond what the current architecture of dtcheck.toolforge.org/dtstats can sustain.
Absent of there being a clear need for the Editing Team to invest in updating dtcheck to cope with the volume of edits people are using the Reply Tool to make, we will sunset the feature on September 1, 2023.
Please comment on T341821 if you see reasons why disabling dtcheck would be disruptive for you.
As of 19 May, a new experience is live for the new comment indicator that appears in the Reply Tool when someone publishes a comment while you are drafting a reply.
The Editing Team would like to acknowledge Beta Kots for the work she did in leading the design of this improvement.
Scaling
On Monday, 7 March 2022, the Reply Tool became available to everyone (logged in and out) on desktop at English Wikipedia (en.wiki). Details about this deployment can be found in Phabricator.
Scaling
As of today, the Reply Tool is available to everyone (logged in and out) on desktop at all Wikimedia wikis except for en.wiki (T296645), fi.wiki (T297533), and ru.wiki (T297410). You can see the full list of what Talk pages project features are available at what wikis by visiting Talk pages project/Deployment Status.
Notifications about New Comments
The team is working on introducing functionality that will alert you, in real-time, when someone posts a new comment in the discussion you are using the Reply Tool within. Instructions for how to try the prototype and share feedback about it can be found here.
Scaling
Today, 7 December, the Reply Tool became available by default on desktop to everyone — logged in and out – at MediaWiki.org. You can try the tool on Talk:Talk pages project/Usability.
Scaling
Today, 17 November, the Reply Tool became available by default on desktop to everyone — logged in and out – at Commons.
Scaling
On Monday, 25 October, the Reply Tool became available by default on desktop to everyone—logged in and out–at the French Wikipedia.
Conversations are ongoing with volunteers about also offering the Reply Tool as an on-by-default feature at the English, Finnish, and German-language Wikipedias.
New functionality
Starting next week, you will gain the ability to preserve text formatting when pasting content into the Reply Tool's Source mode.
Scaling
Today, 30 September, the Reply Tool became available as an on-by-default feature at all Wikimedia Wikis except the projects listed here:
- English Wikipedia
- Finnish Wikipedia
- French Wikipedia
- Russian Wikipedia
- German Wikipedia
- Wikimedia Commons
- Meta-Wiki
- MediaWiki
In the coming weeks, we will start conversations with volunteers at the wikis listed above about the prospect of offering the Reply Tool as an on-by default feature there as well.
In the meantime, you can see how often people are using the Reply Tool to publish talk page comments by visiting https://dtcheck.toolforge.org/dtstats.html.
Scaling
Today, 31 August, the Reply Tool became available as an on-by-default feature at 21 new Wikipedias:
Spanish (eswiki), Italian (itwiki), Japanese (jawiki), Persian (fawiki), Polish (plwiki), Hebrew (hewiki), Dutch (nlwiki), Hindi (hiwiki), Korean (kowiki), Vietnamese (viwiki), Thai (thwiki), Portuguese (ptwiki), Bengali (bnwiki), Egyptian (arzwiki), Swahili (swwiki), Chinese (zhwiki), Ukrainian (ukwiki), Indonesia (idwiki), Amharic (amwiki), Oromo (omwiki), Afrikaans (afwiki).
Usage
To date, people have used the Reply Tool to post 250,000+ talk page comments.
New toolbar in Source mode
Yesterday, 5 August, the Source mode toolbar that was introduced as an opt-in setting in May, became available to all people who have the Reply Tool enabled by default. Note: you can still turn this toolbar off if you would like within Special:Preferences.
Offering the Reply Tool by default at more projects
In the coming weeks, we expect to be able to resume plans with offering the Reply Tool as a default-on feature for all users at all projects. Once the next set of wikis where the Reply Tool will be offered by default is finalized, we will post an update to this page.
For context, plans to offer the Reply Tool more broadly had been stalled while we worked with the Performance and Data Persistence Teams on infrastructure enhancements to ensure the Reply Tool continues to function as expected at this larger scale.
New toolbar in Source mode
As of Tuesday, 11 May, you can enable a toolbar within the Reply Tool's Source mode. This toolbar introduces easier ways for pinging other people and inserting links into the comments you are drafting.
To start, this new mode is being offered as an opt-in preference. You can enable it by taking the following steps
- Ensure you have the Reply Tool enabled
- Visit
Special:Preferences#mw-prefsection-editing - Locate the
Discussion pagessection - "Check" the checkbox next to the setting that reads:
Enable experimental tools in the quick replying and quick topic adding features' source modes - Click
Save - ✅ You are done. You should now see the new toolbar whenever you open the Reply Tool's Source mode
A/B test results
The results are in from the A/B test of the Reply Tool that ran from 11 February through 10 March on 22 Wikipedias. What follows are the conclusions we are drawing from these results and the steps we are taking next.
Conclusions
- The Reply Tool decreases disruption:
- There was a 79.5% decrease in the revert rate of comments Junior Contributors made with the Reply Tool compared to comments Junior Contributors made with page editing.
- There was no significant increase in the number of Junior Contributors who were blocked after publishing a comment with the Reply Tool
- The Reply Tool causes Junior Contributors to have more success publishing comments :
- 72.9% of Junior Contributors who opened the Reply Tool were able to publish at least one comment during the A/B test compared to 27.6% of Junior Contributors who attempted a comment using page editing. This amounts to a 160% increase in the comment completion rate of Junior Contributors who used the Reply Tool during the test than those who used page editing.
- Junior contributors using the Reply Tool are 7.2 times more likely to publish a comment they start than Junior Contributors using page editing.
The full report can be found here: Reply Tool AB Test Report.
Next steps
Considering the A/B test has shown the Reply Tool causes a greater percentage of Junior Contributors to publish a comment without a significant increase in disruption, we will proceed with the plan we shared in T252057 to begin making the Reply Tool available as an opt-out preference at all Wikimedia projects. The sequence and timing of these deployments will be finalized in T280388.
Deployments
23 March, the Reply and the New Discussion Tools will be available as an opt-in beta feature at the German Wikipedia, so it will be at all Wikimedia Sister Projects from 23 March.
A/B Test
On 10-March, the Reply Tool A/B test finished. We are now analyzing the data and expect to have results for you to review in the next four to six weeks.
Deployments
Yesterday, 16-March, the Reply and the New Discussion Tools became available as an opt-in beta feature at the English and Russian Wikipedias and all Wikimedia Sister Projects (except German Wikipedia).
Note: you can review how and where all Talk pages project features are available here: Talk pages project.
A/B Test
On 11 February, an A/B test of the Reply Tool started at 22 Wikipedias (full list here). During this test, 50% of all editors at the Wikipedias included in the test will have the Reply tool automatically enabled, and 50% will not. People at these wikis will still be able to turn the tool on or off in Special:Preferences. The results of this test will help the team decide whether people benefit enough from the tool for it to begin being made available by default, at all projects. We expect to have results to share in 2-3 months.
Engagement metrics
On 5 January 2021, the team completed an analysis of the first ~2.5 months of Reply Tool usage at the three wikis where it is available by default (Arabic, Czech and Hungarian). The findings are encouraging, tho not conclusive considering the limited sample size.
A notable finding from this analysis: the edit completion rate for Junior Contributors using the Reply Tool is 4x higher than the edit completion rate for people using full-page wikitext editing.
You can review these findings in more detail by reading the Metrics section below and/or reviewing the full report: Reply Tool workflow engagement metrics.
Completion rates for comments made with the Reply tool and full-page wikitext editing. Details and limitations are in this report.
A/B Test
We have decided the key metric we will use to compare the A/B test's control and test groups will be the rate at which people in either group publish the comments they initiate.
More information in can be found in the Metrics section below and in Phabricator here: T252057.
It is important to note that we are not considering notable positive changes in the rates at which people start and continue participating on talk pages as well as the rates at which people are participating on talk pages in disruptive ways as prerequisites for the Reply Tool being made available to all people, at all wikis, as an opt-out user preference. We've come to think this for the following reasons:
- Evidence to date suggests the Reply Tool is having the intended short-term impact: Junior and Senior Contributors have an easier time participating in existing conversations.
- We assume notable changes in the rates at which people start and continue participating on talk pages will likely get expressed over a longer period of time (read: they will not be detectable during the course of the upcoming A/B test).
Measuring engagement
The team is analyzing the Reply Tool usage data from the Arabic, Czech and Hungarian Wikipedias, where the tool has been available as an opt-out feature since 24-September-2020.
This analysis will help us determine whether people, across experience levels, are having success using the Reply Tool and whether people are using the tool in ways that disrupts others.
We will post the results of this analysis in the Metrics section below (see: "Analysis 2: Engagement"). You can expect these results to be posted no later than January, 2021.
Evaluating impact
The team is preparing to run an A/B test to evaluate the impact the Reply Tool has had on how people use talk pages. This test will help the team decide whether we should move forward with plans to offer the tool to all people, at all wikis, as an opt-out user preference.
More details about the test can be found in the Metrics section below (see: "Analysis 3: Impact").
Deployment planning
Today, 4-November, the Reply Tool became available as an opt-in Beta Feature at an additional ~250 Wikipedias. This means the Reply Tool is now available as either a Beta Feature or opt-out user preference at all Wikipedias except for the following projects: English, Finnish, Gan, German, Inuktitut, Kazakh, Kurdish, Russian, Tajik, and Uzbek.
Deployment planning
In the coming weeks, we are planning to offer the Reply Tool as an opt-in Beta Feature at all remaining Wikipedias except for the following projects: English, Gan, German, Inuktitut, Kazakh, Kurdish, Russian, Tajik, and Uzbek.
An exact date has not yet been set. Once it has, we will announce it on this page and in Tech/News.
Note: the above was also mentioned in this week's Tech/News. See: Tech/News/2020/44.
Deployment
On Wednesday, 14-October, the Reply Tool became available as an opt-in Beta Feature at an additional 30 projects. The objective of this deployment is to ensure the Reply Tool is usable and useful for editors at Wikipedias that write in languages/scripts with unique characteristics (e.g. long words, unusual font sizing, many diacrtics, rare fonts, etc.).
The full list of projects where the Reply Tool is currently available can be found here: Talk pages project.
Deployment
On Thursday, 24-September, the Reply Tool became available to all users (logged in and out) at the following Wikipedias: Arabic, Czech and Hungarian. This deployment marks the first time the tool is available to people who have not enabled the feature in Beta Features or appended ?dtenable=1 to the URL of the talk page they are viewing.
The primary goal of this deployment is to understand whether contributors, across experience levels, are having success using the feature and finding it valuable. You can find more details about how we think about the deployment process here: Deployment process.
You can see the kinds of edits people at the Arabic, Czech and Hungarian Wikipedias are making with the Reply Tool by filtering Recent Changes:
- Arabic: https://w.wiki/dgV
- Czech: https://w.wiki/dT7
- Hungarian: https://w.wiki/dT6
Custom edit summaries
By the end of Thursday, 17-September-2020, anyone using the Reply Tool on a production wiki will be able to customize the edit summary that accompanies the comments they post with the Reply Tool.
This functionality can be accessed by clicking the Advanced link that will appear beneath the text input area (see the "Custom edit summary implementation" screenshot).
Custom edit summaries
Inspired by the feedback volunteers have shared [i][ii][iii][iv], the Reply Tool will soon offer people the ability to customize the edit summaries that accompany comments posted with it.
As part of implementing the custom edit summary functionality, we created a technical prototype.[1]
You can try the prototype by visiting this page: https://patchdemo.wmflabs.org/wikis/23fd7e0b373b74aceaf8ddec1d82ab09/w/index.php/Talk:Main_Page.
If there are comments and/or questions that the prototype brings to mind, we would value you sharing them on the talk page here: Topic:Vstemnhdi8w8iw95.
Deployment
- On 6-August-2020, the Reply Tool became available as a Beta Feature at the following Wikipedias: Catalan, Chinese, Czech, Georgian, Korean, Serbian, Sorani Kurdish and Swedish.
Metrics
- Earlier this week, the team completed an analysis that looked at how people on our four partner wikis have used the Reply tool between 31-March-2020 and 30-June-2020.
- The key question we were trying to answer with this analysis was: Do the people who have tried the tool find it valuable?
- In summary, we are confident that people who tried the Reply tool do find the tool valuable.
- You can read more detail about this analysis in the Metrics section below.
- Metrics: the team is analyzing how people on our partner wikis have used the Reply Tool between 31-March-2020 and 30-June-2020. You can expect findings to be posted in the Metrics section below in the next two weeks.
- Refinements: in response to the usability tests we ran in June, the team is working on a series of refinements/enhancements to the tool. These are listed below along with the conversations where we are discussing these enhancements:
- Support for custom edit summaries: Topic:Vju7lfcav875rt8r.
- Adjustments to the toolbar: Topic:Vq9yz2a4xa2upy4p
Version 2.0 usability test findings
- To evaluate whether the new features introduced in version 2.0 of the Replying tool are intuitive to Junior and Senior Contributors, we ran two usability tests that asked test participants to complete a series of tasks.
- These test findings are now available for you to review: Version 2.0 prototype (usertesting.com + mediawiki.org).
Version 2.0 deployment
- Today, 17-June-2020, Version 2.0 of the Reply tool became available in the following places:
- Version 2.0 is now available as a Beta Feature on our partner Wikipedias (Arabic, Dutch, French and Hungarian).
- Version 2.0 is now available for testing on all wikis by adding the following parameter to any talk page URL:
?dtenable=1.
Version 2.0 testing and deployment
- Usability tests being run on MediaWiki.org, at usertesting.com and on our four partner Wikipedias (Arabic, Dutch, French and Hungarian) are ongoing.
- Once these tests are complete, findings will be summarized and posted to the Usability testing section below.
Version 1.0 deployment
- Version 1.0 of the tool is now available for testing as a Beta Feature on MediaWiki.org. To turn it on, visit Beta Features in your preferences and make sure the box next to "Discussion tools" is checked (☑️).
Version 2.0 testing
- Version 2.0 of the Reply tool is ready for testing. It introduces functionality for pinging other people by typing @ and writing and styling comments in a new visual mode.
- You can find instructions for how to try the new version of the Reply tool here: Version 2.0 test instructions.
Version 2.0 deployment
- As of today, Version 2.0 of the Reply tool is available to everyone at our partner Wikipedias (Arabic, Dutch, French and Hungarian) via a custom URL.
- To try Version 2.0 of tool at these wikis, add the following text to any URL where the Reply tool is already available:
?dtvisual=1. - Once we have evidence this new version is working as expected, it will become available as a Beta Feature on our partner wikis.
Version 1.0 deployment
- For ~4 weeks, Version 1.0 of the new Replying tool has been available as a Beta Feature on our partner wikis.
- In line with Step 2 of our deployment process, we have been focused on making sure the Reply tool is available on pages where discussions happen outside of talk namespaces. To this end...
- You can now use the Reply tool on any page where the
__NEWSECTIONLINK__syntax is present.- This means people can now use the Reply tool on pages like Dutch Wikipedia's The Pub. You can see the comments people are posting there by reviewing these Recent Changes.
- Within the next week, the Reply tool will be available on in all namespaces where the Signature button is present.
- This will mean you can use the Reply tool on pages without needing the "New section" tab to be available as well. This will be especially helpful on pages like French Wikipedia's Le Bistro, where conversations are transcluded from other pages.
- You can now use the Reply tool to reply to comments transcluded from another page.
- This means it is now possible to use the Reply tool on pages like the Dutch Wikipedia's page for discussing article deletions (Wikipedia:Te_beoordelen pagina).
- You can now use the Reply tool on any page where the
Version 2.0 development
Our two areas of focus right now related to Version 2.0 are:
- Implementing an easy way to @-mention/ping other people.
- Making it possible to switch back and forth between the source and visual text input modes.
Thank you to Pbsouthwood, Pelagic, Samat and TheDJ for the efforts you have made offering design feedback about the version 2.0 mockups.
Version 2.0 designs
- We have designs ready for Version 2.0 of the new Replying tool that we would value your feedback on.
- If you have any thoughts or questions, please post them on the talk page here: Design feedback: version 2.0 mockups.
Version 1.0 deployment
- Today, Version 1.0 of the new Replying tool became available as a Beta Feature on the following partner wikis: Arabic, Dutch, French and Hungarian.
- To ensure the new tool works in ways people expect, we will work together with volunteers at these four Wikipedias over the next couple of weeks to monitor how the feature is being used before deciding deploying it to more people on these as an opt-out user Preference.
- You can learn more about our deployment process below: Deployment process.
Version 1.0 deployment
- Version 1.0 of the Replying feature is currently being tested. Assuming the QA process does not surface any critical issues, the new replying tool will be available as a Beta Feature on the Arabic, Dutch, French and Hungarian Wikipedias as soon as next week.
Version 1.0 deployment
- The deployment of the new replying workflow as a Beta Feature on Arabic, Dutch, French and Hungarian Wikipedias will be delayed for an unspecified amount of time. The team will share an update on this page as soon as a new date is set.
Active
Version 1.0 deployment
- On Monday, 16-March-2020, the team is planning to deploy Version 1.0 of the new workflow for replying to specific comments on talk pages as a Beta Feature on the following Wikipedias: Arabic, Dutch, French and Hungarian.
- You can read more about how this deployment decision was made below: Deploying Version 1.0 as a Beta Feature to partner wikis.
- If there are issues you think should be fixed that have not yet been prioritized, please mention something on the talk page: Talk:Talk pages project/replying. Note: the team is especially interested in links to examples where using the Replying tool disrupts content elsewhere on the talk page. This Phabricator task is a good example of the kind of "disruption" we have in mind: phab:T246481.
Version 2.0 development
- The team continues to develop the new functionality that will be introduced in Version 2.0. Functionality like drafting comments using a rich text/visual editing mode.
Recently completed
- Version 1.0 instrumentation: The team recently completed implementing and testing the instrumentation that will help them evaluate whether contributors are having success using the new Replying feature and whether contributors are using the tool in ways that could be negatively impacting the experience of others.
Active
- Version 1.0 deployment: The team is working to resolve the highest priority issues that surfaced during the initial deployment. Once this is complete, the feature will be deployed as a Beta Feature to four target Wikipedias: Arabic, Dutch, French, and Hungarian. Before this happens, contributors to these wikis will be notified.
- Version 2.0 designs: The team is working on the designs for version 2.0. You can see an early iteration of the designs here: #Version 2.0.
Recently completed
- Version 1.0 deployment: on Tuesday, 18-February, Version 1.0 of the feature became live via a custom URL for testing in production, on four Wikipedias: Arabic, French, Dutch and Hungarian.
- We, along with many of you (thank you!), have been testing the feature extensively to make sure it appears where it should and behaves how we expect. You can read and add to the conversations where people are sharing feedback in these places: MediaWiki, Arabic Wikipedia, Dutch Wikipedia, French Wikipedia and Hungarian Wikipedia.
- Deployment status and process: the team published information about why, how and where the Replying feature is and will be deployed here: #Deployment.
Active
- Version 1.0 deployment: this week, the team will deploy version 1.0 of the new replying workflow to the first set of production Wikipedias (Arabic, Dutch, French and Hungarian). For the time being, the new feature will only be accessible using a custom URL. This way, the team can safely test the feature without impacting anybody else's experience. If this testing goes well, the feature will then be deployed as an opt-in Beta Feature.
- Version 2.0: the team is working on the design of the next iteration of the feature. If you would like to review and add your feedback on the latest design, please review this Phabricator task: T235593.
Recently completed
- Finalized the list of features that will be included in the next iteration of the feature. You can review the list of features that are planned here: #Versions.
Active
- Finishing development of version 1.0 ahead of the team's planned deployment of the new replying workflow to four wikis, beginning as an opt-in Beta Feature, in mid- to late-February.
- Adding instrumentation to the new replying workflow. This will enable the team to measure how contributors are engaging with the workflow once it is deployed and determine whether adjustments need to be made to make it more intuitive.
Recently completed
- Finalized the user interface for version 1.0 of the feature, including how real time previewing will look and function. You can see a mockup for what it will look like once version 1.0 of the feature is deployed here: File:New replying mockup (v1.0).
Active
- Prototype: the team is running an on-wiki user test of version 1.0 of the prototype. We would value you trying it out! To try the prototype and participate in the test, please visit: Prototype version 1.0 usability test
- Measurement: the team is drafting a plan for how we intend to measure the quantitative impact of the improved workflow for replying to specific comments.
Recently completed
- User testing: the team recently completed two user tests. Both tests involved people familiar with Wikipedia and who have little or no experience participating on Wikipedia talk pages. The first test was meant to identify the challenges people face when trying to participate on talk pages using the existing experience. The second test is meant to identify usability issues with version 1.0 of the prototype for the improved workflow for replying to specific comments. Read more about these tests below: User testing.
Objectives
This work is intended to make participating productively on talk pages easier and more intuitive for contributors.
"Easier" means more experienced contributors can participate in existing discussions with less effort, and "intuitive" means newer contributors do not need specialized knowledge to figure out how to add their thoughts to a conversation. Said in a different way: as a result of this work, both newer and more experienced contributors should report the workflow for participating in existing discussions to be "obvious" and "approachable."
It is important to note the mention of "participating productively" in the first paragraph of this section. We appreciate it is not enough to build tools that simply encourage contributors to "talk more"; they need to encourage people to work together to help improve the project they are discussing.
While we are still defining how to measure the impact of this feature, a key part of this work will involve figuring out how to understand the impact this new affordance has had on the quality of discussions on the talk pages it is deployed to.
Impact
This section discusses the three quantitative analyses we have planned to evaluate the usefulness, usability and impact of the Reply Tool. These analyses are discussed in detail below.
Analysis 3: Impact
In this last analysis, we sought to learn two things:
- Does the Reply Tool cause Junior Contributors to be more successful publishing comments on talk pages?
- Does the Reply Tool cause more disruption to other volunteers?
To answer the questions above, we ran an A/B test of the Reply Tool from 11 February through 10 March on 22 Wikipedias. The results from this test can be found in the "Findings" section below.
Timing
This analysis was completed on 23-April-2021. The A/B test ran from 11-Febraury-2021 through 10-March-2021.
Conclusions
The clear evidence that shows the Reply Tool causes a greater percentage of Junior Contributors to publish a comment without a significant increase in disruption, signals to the team that it would benefit all Wikimedia Projects and volunteers to have the Reply Tool made available as an opt-out preference.
Findings
The data below included logged-in users who had not previously interacted with the Reply Tool (defined as users whose discussiontools-editmode preference was empty).
- 72.9% of Junior Contributors who had access to the Reply Tool were able to publish at least one comment during the A/B test compared to 27.6% of Junior Contributors who did NOT have access to the Reply Tool.
- Junior contributors were 7.2 times more likely to publish a comment than junior contributors using page editing.
- There was a 79.5% decrease in the revert rate of comments Junior Contributors made with the Reply Tool compared to comments Junior Contributors made with page editing.
Report
You can review the full analysis results here: https://wikimedia-research.github.io/Reply-tools-analysis-2021//.
Analysis 2: Engagement
The next analysis we did was needed to help determine whether people, across experience levels, were having success using the Reply Tool and whether people were using the tool in ways that degrades the experiences of others.
This analysis's findings were used to decide whether the Reply Tool is functioning well enough for its impact on user behavior to be tested via a larger-scale (read: at wikis beyond our partner wikis) A/B test.
Timing
This analysis was completed on 5-January-2021
Conclusions
The high rate at which people who used the Reply Tool to publish the comments they started writing, combined with the lack of clear evidence of disruption, led the team to think the Reply Tool was ready to be tested at a larger scale via an A/B test. More details below.
Findings
- The edit completion rate for Junior Contributors using the Reply Tool is 4x higher than the edit completion rate for people using full-page wikitext editing. Source
- People who have made <1,000 cumulative edits, are between 32% and 510% more likely to publish edits with the Reply Tool (across modes) than they are with full page wikitext talk page editing. Source
- <1% of comments Junior Contributors post with the Reply Tool are reverted within 48 hours, across wikis (this number is higher for Senior Contributors ). Source
Analysis 1: Adoption
In this first analysis, we sought to learn whether people at the Arabic, Dutch, French and Hungarian Wikipedias, where the Reply Tool had been available as a Beta Feature since 31-March-2020, were finding the tool valuable.
To determine the extent to which people were "finding the tool valuable" we looked at how frequently people were using the tool (as measured by the number of distinct days they use it on) and how intensely people were using the tool (as measured by the total number of edits they make with the Reply Tool and the percentage of total talk page edits they used the Reply Tool to make).
Understanding the above helped us determine the tool was ready to be made available as an opt-out feature at these wikis and made available as an opt-in Beta Feature at others.
Timing
This analysis was completed on 28-July-2020.
Findings
The data below included people who have used the Reply Tool as a Beta Feature at the Arabic, Dutch, French and Hungarian Wikipedias, between 31-March and 30-June-2020:
- 328 people posted at least one comment using the Reply Tool
- 69.5% of people posted ≥ 2 comments with the Reply Tool
- 61.3% of people posted a comment using the Reply Tool on ≥ 2 distinct days[2]
- 23.6% of people used the Reply Tool to make over half of their talk page edits[3]
Report
You can review the full analysis results in this Jupyter notebook: https://nbviewer.jupyter.org/github/wikimedia-research/Reply-tools-analysis-2021/blob/master/Adoption-Metrics/Replying-Tool-Adoption-Metrics.ipynb.
Background
This year, the Editing Team is committed to improving how contributors communicate and collaborate on Wikipedia, using talk pages.
For talk pages to be valuable, contributors need to intuitively know how to participate in the conversations that happen on them.
The trouble is – as previous user testing, the Talk Page Consultation and the team's research uncovered – contributors, across experience levels, find replying to specific comments on Talk pages to be challenging.
Specifically, the team's research has found:
- More experienced contributors find it difficult to locate the specific comment they are wanting to reply to when participating in long, multi-person conversations, within big blocks of wikitext.
- Newer contributors report not being sure how to reply to a conversation, regardless of its length or complexity.
In exploring an affordance for replying to specific comments on talk pages, we are striving to make participating productively on talk pages easier and more intuitive. We think doing so will help newer contributors understand and use talk pages as places to communicate with others and help more senior contributors collaborate more efficiently.
Challenges
One part of building new features is codifying social conventions in software. In this context, "conventions" could mean deciding what character the software should use for automatically indenting or outdenting replies. "Conventions" could also mean deciding how the software should represent the first comment in a discussion in wikitext: Is there a linebreak between the reply and the original comment? Is the reply added to the line immediately following the original comment in the discussion? Is there another way this comment should be represented?
It is unlikely all communities will answer these questions in the same way. This means the software needs to be built in a flexible enough way to accommodate these different preferences. This is possible, although it adds complexity to the design and development processes.
Design
To increase the likelihood this enhancement is useful for contributors across experience levels, we have broken down the improvements we have planned into a series of releases. These different releases are outlined in more detail in the "Versions" section below.
Also below are the latest iteration for the designs we are planning to implement in Version 1.0. If you have thoughts about anything included in this section, we would value you sharing them on the talk page: Talk:Talk pages project/replying.
❗️Please consider the features included below as drafts and expect them to evolve as we learn new things.
Versions
Version 1.0
.png)
This version will introduce the basic reply functionality to validate the core workflow. This version will likely include the following features:
- An affordance for replying to specific comments
- A text box for composing replies using wikitext
- A way for signatures to be appended to comments, automatically
- A way for those comments to be indented or outdented, automatically
You can try a prototype of version 1.0 of the new replying workflow here: https://en.wikipedia.beta.wmflabs.org/w/index.php?title=Talk:Cats.
A mockup showing how version 1.0 will likely look once it is deployed can be seen in this screenshot: File:New replying mockup (v1.0).png
Version 2.0
Assuming version 1.0, and any smaller releases that follow, helps us to validate and refine the core replying functionality, version 2.0 will likely include enhancements to make it easier and intuitive for Junior Contributors to draft and post their replies. This version will likely include the addition of the following features:
- Opt-in/out of watching the talk page as part of the replying workflow
- Populating the reply text input with instructional text to make it clear who you are responding to.
- Drafting and formatting replies using a rich text editor
- Switching between "rich text" and "source" writing modes while preserving any content contributors have already written
- Mentioning/notifying specific users in a conversation without needing to know about or interact with wikitext
- Editing a specific comment
- Note: It is not likely editing a specific comment will be included in the initial release of version 2.0. This is due to the technical complexity involved with implementing this functionality. You can learn more about this "technical complexity" and the initial implementation we have planned in Phabricator here: T245225.
-
Version 2.0 overview: a mockup showing how version 2.0 of the new replying workflow could look. -
Version 2.0. (visual): an annotated mockup showing the new Replying feature's visual text input mode. -
Version 2.0 (source): an annotated mockup showing the new Replying feature's source (read: wikitext) text input mode.
Deployment
This section contains information about how and where the Replying feature is and will be deployed.
Deployment status
To see what projects the Reply Tool is available, please visit: Talk pages project.
Deployment decisions
Deploying Version 1.0 as a Beta Feature to partner wikis
On Monday, 16-March-2020, the team is planning to deploy Version 1.0 of the new workflow for replying to specific comments on talk pages as a Beta Feature on the following Wikipedias: Arabic, Dutch, French and Hungarian.
This decision is the outcome of the team being confident in the following:
- Reply links are available in the places (read: talk pages, across namespaces) where people expect them to be.
- The core parts of the Replying workflow (e.g. writing a comment, previewing a comment, formatting a comment, discarding a comment and publishing a comment) are working in the ways people expect.
- The Replying workflow is not disrupting existing talk page content.
This confidence is based on the team and volunteers testing the new Replying workflow on >75 different talk pages in three talk namespaces on the 4 Wikipedias where the feature will be deployed.
This testing proved necessary for identifying and resolving issues related to the core parts of the replying workflow. Issues like:
- The replying text box remaining after cancelling a reply: phab:T245574.
- Reply links appearing after list items that were not comments: phab:T245692 (thank you, User:Bdijkstra).
- Reply links not appearing after comments that contained signatures with an invisible Unicode character: phab:T245784.
- The reply tool not working in Safari: phab:T245781.
- The reply tool removing a page from your watchlist: phab:T245579 (thank you, User:Tacsipacsi).
- Broken and incomplete table syntax causing page corruption: phab:T246481.
This testing was also important for identifying additional use cases the team is actively working to provide support for. Use cases like being able to reply to comments transcluded from a subpage. For more details on this work, see this Phabricator task: phab:T245694.
Deployment process
The deployment of this feature will happen in four steps. These four steps are described below.
Before the team decides to move on to the next step in the deployment process, they will do two things:
- Make contributors affected by the next planned deployment aware of where to test the version of the feature the team is planning to deploy, how the feature will be deployed and what informed the deployment decision. Announcements containing this information will be posted as updates on this project page and publicly on all wikis affected by the planned deployment.
- Invite contributors to share their input about the next deployment the team is planning. Contributors will be invited to share their input about the next deployment on the talk page and on pages where the deployment is announced.
Step 1: Beta cluster
Deployment goal
The goal of this step in the deployment process is to answer these questions:
- Do contributors find replying to comments using the feature intuitive?
- Are any parts of the workflow broken (e.g. writing and formatting a reply, cancelling a reply, previewing a reply, publishing a reply)?
Deployment details
To make sure the feature behaves in ways contributors expect, new versions of the feature will be deployed to the Beta Cluster. This way, contributors will be able to safely experiment with new versions of the Replying feature without needing to worry about affecting existing content or contributors.
Step 2: Beta Feature
Deployment goal
The goal of this step in the deployment process is to answer these questions:
- Is the feature available where it should be?
- Does the feature affect existing talk page content in unintended ways?
- Are there any issues that prevent contributors from being able to respond to comments?
Deployment details
To make sure the feature behaves in ways contributor expect on production wikis before exposing the feature to a large number of people, the feature will be deployed as a Beta Feature.
Step 3: User Preference (opt-out)
Deployment goal
The goal of this step in the deployment process is to answer these questions:
- Are people able to successfully post replies using the feature?
- Is the feature having any unintended affects (e.g. encouraging disruptive talk page comments edits)?
- Are people finding the feature valuable?
Deployment details
To determine whether a broad range of people are having success using the feature and are finding it valuable, the feature will be deployed as an opt-out User Preference.
Step 4: A/B test
Deployment goal
The goal of this step in the deployment process is to understand whether the feature had the impact it was intended to have. To determine this, we will use this deployment step to answer:
- Does the new replying feature make it easier and more intuitive for Junior and Senior Contributors to respond to specific comments on Wikipedia talk pages?
Deployment details
To determine whether the feature had the impact it was intended to have, the team will run an A/B test on a to-be-determined set of wikis.
The team is finalizing the details of this A/B test.
Step 5: All wikis
If evidence from the previous 4 deployment steps suggests the new Replying feature makes replying to specific comments on talk pages easier and more intuitive for Junior and Senior Contributors, it will be deployed to all remaining Wikipedias. Determining how and when the feature is deployed depends on the team, along with volunteers from those wikis, being confident the feature works reliably and in ways people expect it to.
Usability testing
This section contains information about user testing the team conducts to ensure the revised experience works in ways contributors expect.
Version 2.0 prototype
To evaluate whether the new features introduced in version 2.0 of the Replying tool are intuitive to Junior and Senior Contributors, we ran two usability tests that asked test participants to complete a series of tasks:
- One test recruited Junior Contributors (read: people familiar with reading Wikipedia and who have little-to-no experience using Wikipedia talk pages) via usertesting.com.
- One test recruited Senior Contributors (read: people who have experience using Wikipedia talk pages) via MediaWiki.org.
Overall, these tests demonstrated the majority of Junior and Senior Contributors were successful in using Version 2.0 of the Reply tool to write, format and publish comments on Wikipedia talk pages.
You can learn about the overall impressions people had, the key areas for improvement and next steps below.
Overal impressions
Senior Contributors described the tool as well-balanced and expressed appreciation for the new automatic pinging feature. Here are a few noteworthy comments:
- "I can insert a link to an [sic] user page inserting an @, after which a list of usernames pops up. That is nice."
- "...this [pinging] is a game changer."
- "Nice tool for both beginners and experienced users."
- "Good tool, simple to use, it's good for what it's made for."
Junior Contributors who were able to find the Reply tool successfully completed the tasks they were asked to and described the tool as being intuitive and straightforward to use.
With this said, when peoples' focus moved from the Reply tool to the broader talk page, they became confused and uncertain about "where" they were and what they ought to do.[4] Here are a few comments we thought were noteworthy:
- "It was pretty straightforward. streamlined process here i like the fact that it's very stripped down and doesn't feel data intense.so not overwhelming by any stretch of the imagination. the colors are pleasing to the eye... simple to navigate interface. Overall it works the way i'd expect it to."
- "I'm a little confused about what User talk: Alice is..."
- "Doesn't look like the sort of usual messaging interface you might see on a website."
Task analysis
The table below shows whether participants in each test group were able to complete the following tasks with Version 2.0 of the Reply tool.[5]
You can review the broader themes these tests helped our team identify in the "Key themes" section below.
| Task description | Junior Contributors | Senior Contributors |
|---|---|---|
| Identify the way to reply to a comment on the talk page | ✅9/10 were successful | ✅13/13 were successful |
| Type a comment using the new visual mode | ✅9/10 were successful | ✅13/13 were successful |
| Format the content they had written using the visual mode's formatting tools | ✅8/10 were successful | ✅13/13 were successful |
| Ping someone who has already commented in the section they are commenting in | ⚠️ 6/10 were successful | ✅10/13 were successful |
| Ping someone who has not already commented in the section they are commenting in | ⚠️ 7/10 were successful | ✅10/13 were successful |
| Remove one of the pings they created | ✅8/10 were successful | ⚠️9/13 were successful |
| Review how the comment they wrote will appear in wikitext before publishing | Not asked | ✅13/13 were successful |
| Publish the comment they wrote | ✅9/10 were successful | ✅13/13 were successful |
| Locate the comment they published | ✅9/10 were successful | ✅13/13 were successful |
Areas for improvement
These are the broader areas for improvement that surfaced through these usability tests. You can review how we plan to address these themes in the "Next steps" section below.
- The tool's two modes: visual and source:
- Some Senior Contributors were surprised to learn when viewing a diff that the visual mode automatically signed the comment they posted.
- Some Junior Contributors did not find the difference between the visual and source modes to be clear.
- Pinging
- Some Senior Contributors said they would not have known the pinging feature was available had they not been instructed to try it.
- Some Senior Contributors wondered whether Junior Contributors would be confused by the normal link inspector that is presented when a ping within a drafted comment is selected.
- Commenting in long threads
- Senior Contributors found it can be easy to "lose the tool" when trying to comment in a conversation and/or on a page with many comments.
- An example: when trying to respond to a comment that has many existing replies, it can be difficult to simultaneously look at the comment you are responding to and the comment you are drafting using the Reply tool.
- Senior Contributors found the tools within the visual mode hard to "reach" when working on a large screen and when writing a lengthier comment.
- Senior Contributors found it can be easy to "lose the tool" when trying to comment in a conversation and/or on a page with many comments.
- Edit summaries
- Some Senior Contributors wondered whether supports for custom edit summaries would be added.
Next steps
To address the issues and opportunities these two usability tests surfaced, the team will be working on improvements to make the following come true;
- Contributors, across experience levels, can easily mention/ping another person in the comment they are writing.
- Example improvement: adding an icon to the visual mode's toolbar that communicates there is a feature for pinging people.
- Contributors, across experience levels, know how the tool's two input modes function and how they relate to one another.
- Example improvement: revising what the visual / source modes are called and how prominently they appear in the interface.
- Contributors, across experience levels, find the tool well suited for commenting in talk page conversations, regardless of how many conversations (read: sections) and comments they contain.
- Example improvement: revising where the text input area appears on the page and how it behaves when the page is scrolled.
You can track the progress we make on the areas above by watching this page and/or by reading this Phabricator workboard: phab:Reply tool version 2.0.
Version 1.0 prototype
To see how the version 1.0 prototype affected Junior Contributors' experience replying in conversations on talk pages, the team ran a control test with 5 participants on usertesting.com. You can review the test findings below.
How were we testing?
This usability test was run on usertesting.com with 5 participants who were each screened to ensure they were technically advanced web users who have used Wikipedia before.
In order to compare the revised replying workflow to the existing workflow, each test participant was asked to complete the same tasks on a desktop computer, while narrating their experience:
- Navigate to an existing discussion happening on a test wiki talk page
- Draft and publish a response in that discussion
- Locate their published response on the talk page
- Navigate to the talk page's history page and locate the reply they had just posted
- Start drafting another reply, but discard it before publishing
What did we find?
Overall, the prototype seems to have improved Junior/newer contributors' experiences replying in an existing conversations. On average, it took participants using the prototype half the time to publish a reply compared to the time it took them in the previous test, using full page editing.
Task completion
- ✅All participants were able to locate the specific discussion on the talk page they were being instructed to reply to
- ✅All participants successfully composed and published a reply in an existing discussion
- ✅All participants properly indented and signed their comments
- Note: the tool does this for them automatically.
- ✅All participants were able to locate their published response on the talk page
- ✅All participants were able to find the talk page history page
- ❌1 out of 5 participants was able to quickly locate the edit they had just made on the talk page's history page
- ✅All participants were able to cancel their second reply before publishing it
Results
The prototype seems to have made it easier for Junior/newer contributors to reply to existing conversations on talk pages. Test participants used phrases like, "straight-forward," "no-problem whatsoever," and "really easy" to describe their experiences.
With this said, there are still parts of the replying workflow participants found difficult that could be improved:
- Making it easier to visually distinguish between different replies in a discussion
- Making it easier to identify who the author of a comment is and who they are responding to
- Making the "Reply" call to action easier to discover
- Revising the automatically generated edit summary to make it easier for contributors to identify their reply on the History page.
More details can be found in this ticket on Phabricator: T236921#5744471.
Next steps
The team will decide if and when to make the improvements mentioned above (e.g. making it easier to visually distinguish between different replies in a discussion) and do further testing with more experienced contributors (this has started here: Version 1.0 prototype test).
Existing reply experience
To identify the challenges Junior Contributors face when trying to participate in conversations on talk pages, the team ran a control test of the existing (full page) editing workflow.
How were we testing?
This usability test was run on usertesting.com with 5 participants who were each screened to ensure they were technically advanced web users who have used Wikipedia in some capacity before.
Each test participant was asked to do the following tasks on a desktop computer, while narrating their experience:
- Navigate to an existing discussion happening on a test wiki talk page
- Draft and publish a response in that discussion
- Locate their published response on the talk page
- Start drafting another reply, but discard it before publishing
- Navigate to the talk page's history page and locate the reply they had just posted
What did we find?
Below is a summary of our findings from this user test. More details can be found in this ticket on Phabricator: T239175#5723843.
Task completion
- ✅All participants were able to locate the specific discussion on the talk page they were being instructed to reply to
- ✅All participants successfully composed and published a reply in an existing discussion
- ❌0 out of 5 participants properly indented and signed their comments
- ❌1 out of 5 participants noticed they had not signed their comments
- This one person remarked, "I probably should have put my user name, but i didn't".
- 3 out of 5 participants were able to locate their published response on the talk page.
- Note: it took several participants multiple minutes to do this
- ✅4 out of 5 participants were able to find the talk page history page and locate the reply they had just posted
- ✅4 out of 5 participants were able to cancel their second reply before publishing it
Results
This test highlighted an important tension many Junior Contributors seem to face: they finish the task they set out to complete without realizing they might have done so incorrectly. And if they do realize they have made a mistake, they are not equipped to fix it because the proper conventions are not intuitive enough for them to understand.
Next steps
The team will be doing two things in response to this test:
- The team will be doing additional testing in different namespaces and languages (see: Partner wikis).
- We will run a usability test of the prototype with contributors, across experience levels, to see how it affects their experiences replying to comments on talk pages.
History
Many projects have, and are, working to improve contributors' experiences with talk pages. This project is better off for their existence. Some of the projects the team continues to learn from are listed on the main project page and below. If there is a project you think we should be aware of, please boldly add it here.
Glossary
The Talk pages project glossary is intended to help us all communicate about talk pages more effectively by making sure we have a shared understanding about the words we use in our discussions and documentation throughout the project.
See also
- https://quarry.wmflabs.org/query/48614 – Counting the number of times editors use the Reply tool
- ↑ The prototype linked above implements – what we currently understand to be – the core custom edit summary functionality. It intentionally is not opinionated about interface polish/presentation.
- ↑ A "distinct day" was defined as a distinct calendar day. Therefore, some of these edits may have occurred 24 hours apart and some may have occurred only a few hours apart depending on what time of day the edit was made.
- ↑ This data is a bit noisy in so far as there are some edits that cannot be made with the Reply Tool (e.g. starting a new section, editing existing content, etc.) and the software is not currently able to differentiate between comments and other talk page edits.
- ↑ Junior Contributors not recognizing talk pages as places to communicate with other editors is a finding that continues to surface[1] and something we plan to address later in the Talk pages project.
- ↑ These test findings were compiled on 10-June-2020.