Mobile talk page edit completion rate (by account status)
In this impact analysis of the mobile Reply and New Topic Tools, we sought to answer two primary questions:
After the mobile Reply and New Topic Tools were introduced, how – if at all – did the likelihood that people would successfully publish the new talk page topics and comments they started drafting change?
After the mobile Reply and New Topic Tools were introduced, how – if at all – did the likelihood that people would publish talk page edits that are reverted within 48 hours of being published?.
To answer the questions above, we compared the rates at which people published the edits they started and the rates at which published edits were reverted before and after the mobile Reply and New Topic Tools were made available at the following Wikipedias: Arabic, Bengali, Chinese, French, Hebrew, Japanese, Haitian Creole, Korean, and Vietnamese.
The results from this test can be found in the "Findings" section below.
Timing
This analysis was completed on 1 September 2022.
These results include data from the two weeks before and after 29 June 2022.
Mobile talk page edit completion rate (by experience level)
Findings
Edit Completion Rate Increased
There was a 53.4% (12 percentage points) increase in the rate at which people who are logged in and logged out published the mobile talk page edits they started.
There was a 155% (14 percentage points) increase in the rate at which people who have made <100 cumulative edits published the mobile talk page edits they started.
At the 15 Wikipedias that participated in the test, 50% of people who were logged in were shown the new talk page design and 50% were shown the existing experience on desktop talk pages.
We then compared how these two groups of people engaged with talk pages in service of making the following decision: Are the set of Usability Improvements on desktop fit to be made available to everyone, at all Wikimedia wikis, by default?
What follows is a summary of what we learned through this A/B test and ultimately, the conclusion we are drawing from it.
This set of desktop talk page design changes resulting in increased engagement and decreased disruption is leading the Editing Team to be confident that offering these Usability Improvements as default-on features would have a net positive impact on wikis and the people who contribute to them.
Revert rate for talk page edits in A/B test, grouped by experience level.
Revert rate.
Defined at the proportion of published talk page edits, of all types, that are reverted within 48 hours.
Overall: we observed an 11% decrease (-0.4 percentage points) in the revert rate of talk page edits by people shown the suite of desktop talk page design changes across all participating Wikipedias and all editor experience levels.
Experience level: when we analyzed revert rate by editor experience level, we observed...
A 12.5% decrease in the revert rate of talk page edits made by Junior Contributors (<100 cumulative edits)
A 7.4% decrease in the revert rate of talk edits made by Senior Contributors (>100 cumulative edits).
Desktop skin: when we analyzed revert rate by desktop skin type, we observed...
A 6.3%increase (3.2% → 3.4%) in the revert rate of edits made by people using the Vector skin
A 30.8% decrease (3.9% → 2.7%) in the revert rate of edits made by people using the Vector 2022 skin
Edit completion rate
Defined as the proportion of talk page edits that are started and successfully published (not reverted within 48 hours).
Overall: People shown the suite of desktop talk page design changes were slightly more likely to complete an edit that they started. We observed a 3.3% increase (+2 percentage points) in edit completion rate across all participating Wikipedias and all editor experience levels.
Experience level: when we analyzed edit completion rate by experience level, we observed...
A 1.6% increase (43.8%→44.5%) in edit completion rate among Junior Contributors (<100 cumulative edits)
A 3.6% increase (66.8%→69.2%) in edit completion rate among Senior Contributors (>100 cumulative edits)
Desktop skin: when we analyzed revert rate by desktop skin type, we observed...
A 2.1%increase (62.4% → 63.7%) in the revert rate of edits made by people using the Vector skin
A 5.3% decrease (59.9% → 63.1%) in the revert rate of edits made by people using the Vector 2022 skin
Percent of talk page views by Junior Contributors that included an edit attempt
Block rate
Defined as the proportion of people who publish talk page edits and are subsequently blocked.
We did not observe any significant changes in the number of users blocked after publishing a desktop talk page edit.
Overall: 0.5% of users shown the suite of desktop talk page design changes were blocked compared to 0.6% of users who were shown the existing experience across all participating Wikipedias and all editor experience levels.
Talk page engagement
Proportion of Junior Contributors that visited a desktop talk page and made an attempt to edit.
Of all the visits to desktop talk pages that Junior Contributors made throughout the A/B test, Junior Contributors shown the suite of desktop talk page design changes interacted with the page in 16.7% more talk page views (+1.4 percentage points) than those shown the existing experience.
We observed an 19% increase (+0.7 percentage point) in the proportion of talk page views by Junior Contributors that included a saved edit.
Timing
This analysis was completed on 27 June 2023 and analyzed how people engaged with desktop talk pages at az.wiki, bn.wiki, de.wiki, fa.wiki, hi.wiki, he.wiki, id.wiki, ko.wiki, nl.wiki, pt.wiki, ro.wiki, th.wiki, tr.wiki, uk.wiki, and vi.wiki from 8 May 2023 through 28 May 2023.
To learn whether the suite of desktop talk page design changes were negatively impacting peoples' experiences using Wikipedia talk pages, we conducted an initial analysis at the three Wikipedias where these features were first made available by default: ar.wiki, cs.wiki, and hu.wiki.
This analysis was meant to help us answer the following question:
Are the set of Usability Improvements negatively impacting other peoples' experiences using Wikipedia talk pages?
The answer to this question was a key factor in helping the team decide whether these new features were fit to be deployed more widely and evaluated through an A/B test.
Timing
This analysis was completed on 12 April 2023 and analyzed engagement data at ar.wiki, cs.wiki, and hu.wiki during the following time periods:
cs.wiki and hu.wiki: 28 February 2023 - 29 March 2023
ar.wiki: 05 March 2023 - 3 April 2023
The time periods were selected to collect engagement data two weeks before and two weeks after making the suite of desktop talk page design changes available by default at the participating wikis.
Findings
Desktop talk page revert rates by experience level at ar.wiki, cs.wiki, and hu.wiki during spring 2023
Revert rate
Overall: We did not observe any changes in the overall desktop talk page edit revert rate pre- and post-deployment across the three participating wikis. The overall revert rate pre- and post- deployment was 4.3%.
Editor Interface: When we analyzed the revert rate by editor interface type, we observed a -17% percent decrease in the revert rate of desktop talk page edits made with visual editor and a -4% decrease in the revert rate of desktop talk page edits made with wikitext following deployment of the feature.
Experience level: When we analyzed the revert rate by experience level, we observed a -18% decrease in the revert rate of desktop talk page edits by Junior Contributors (under 100 cumulative edits) following the deployment of the feature; compared to 4.5% increase (+0.1 percentage points) for Senior Contributors (over 100 cumulative edits).
Desktop talk page edit completion rates by experience level at ar.wiki, cs.wiki, and hu.wiki during February and March, 2023.
Edit completion rate
Overall: We observed a -4.8% decrease in desktop talk page edit completion rate across all three participating wikis. There were no sudden increases or decreases in daily edit completion rates following deployment of the feature.
Editor Interface: We observed a slight increase (< 1%) in edit completion rate for desktop talk page edits made with visual editor and an -11% percent decrease for desktop talk page edits made with wikitext following deployment of the feature.
Experience level: We observed was a 10.2% increase in talk page edit completion rate for Junior Contributors and a slight decrease (-1.3%) in edit completion rate for Senior Contributors.
Block rate
Overall: We did not observe any significant changes in the number of users blocked after publishing a desktop talk page edit. Less than 1.5% of users were issued a sitewide block pre and post deployment across all three participating wikis.
Opt-out rate
Overall: A total of 48 distinct users across all three participating wikis turned the feature off.
Activity level: Less than 1% of the total number of users who made a desktop talk page edit attempt following the deployment of the usability improvements (2,122 people) disabled the feature during the reviewed time period.
In the A/B test of Topic Subscriptions, we sought to answer three primary questions:
Do Topic Subscriptions cause contributors, across experience levels, to respond more quickly to new topics and comments that are posted on wikitext talk pages?
Do Topic Subscriptions cause contributors, across experience levels, to receive responses to a greater percentage of the comments they post and new topics they start?
Do Topic Subscriptions create unwanted noise that discourages people from using talk pages?
To answer the questions above, we ran an A/B test of Topic Subscriptions on desktop at 20 Wikipedias. The results from this test can be found in the "Findings" section below.
Timing
This analysis was completed on 5 August 2022. The A/B test ran from 2 June through 18 July 2022.
Findings
Topic Subscriptions increase the likelihood that volunteers who have made < 500 cumulative edits will receive a response to the comments they post.
There was a 8.6% increase in the percent of comments posted by people who have made under 100 cumulative edits that received a response
There was a 15.3% increase in the percent of comments posted by people who have made 100-500 edits that received a response
People who had Topic Subscriptions enabled respond to comments 57% faster (51 minute decrease in median response times) than people who did not have access to Topic Subscriptions
Topic Subscriptions reduce long lags in interaction
Among people who had access to Topic Subscriptions during the A/B test, 0.4% of responses were published >10 days after the initial comment was posted
Among people who did NOT have access to Topic Subscriptions during the A/B test, 7.3% of responses were published >10 days after the initial comment was posted
Topic Subscriptions did not cause a sharp increase in the number of notifications people receive each day
Prior to the A/B test, people at the participating Wikipedias received, on average, 3.6 notifications per day across
During the A/B test, people at the participating Wikipedias received, received, on average, 4.2 notifications per day
Topic Subscriptions did not cause fewer Senior Contributors to participate on talk pages.
The number of distinct topics people are actively subscribed to.
In this first analysis, we sought to learn the extent to which Senior Contributors are trying Topic Subscriptions (Manual and Automatic) and finding them valuable.
This information is necessary for the team to decide whether the impact of the feature is ready to be analyzed through an A/B test that is being planned in T280897.
Meta
Time Period Analyzed: 19 October 2021 – 1 March 2022
28% of all people (40% of Senior Contributors) that edited a talk page and have the DiscussionTools beta feature enabled have subscribed to at least 1 topic since 19 October 2021 (when manual topic notifications was available as a beta feature to all Wikis)
The majority (65%) of people actively subscribed to manual topic subscriptions are subscribed to 1 to 2 topics. 35% of manual topic subscribers have subscribed to more than 2 distinct topics.
Value
People value having the option to subscribe to topics: 99.5% of people who who received a new comment notification after manually subscribing to a discussion kept the feature enabled.
People read the new comment notifications they receive: people read 96% of the new comment notifications they receive within two weeks of receiving them.
No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.
Some people are using Topic Subscriptions a lot: while the majority of people (65%) who have manually subscribed to a topic have subscribed to 1 to 2 distinct topics, 6% of people are actively subscribed to more than 20 topics.
Disruption
No sustained increases in the number of notifications sent/person/day: The average number of notifications sent per day has remained fairly stable with a daily average of about 4 notifications per user per day.
In the A/B test of the New Topic Tool, we sought to learn two things:
Does the New Topic Tool cause a greater percentage of Junior Contributors to publish the new topics they start without a significant increase in disruption?
Does the New Topic Tool cause Junior Contributors to be more likely to start (activation) and continue (retention) participating on Wikipedia talk pages?
To answer the questions above, we ran an A/B test of the New Topic Tool from 27 January 2022 through 25 March 2022 on 20 Wikipedias. The results from this test can be found in the "Findings" section below.
Timing
This analysis was completed on 13 April 2022. The A/B test ran from 27 January 2022 through 25 March 2022.
Findings
The New Topic Tool causes logged-in Junior Contributors to have more success publishing topics:
The percentage of Junior Contributors and Senior Contributors that attempted to start a new topic and were successful in publishing at least one of these new topics during the A/B test.Junior Contributors who open the New Topic Tool are ~1.3 times more likely to successfully publish a new topic than Junior Contributors who open the existing add new section workflow.
44.2% of Junior Contributors that opened the New Topic Tool were able to successfully publish at least one new topic during the A/B test compared to 37.2% of Junior Contributors using the existing add new section workflow (+ 7 percentage points; 19% increase).
The New Topic Tool causes logged-in Junior Contributors to be less disruptive:The rate at which the edits Junior Contributors make with the New Topic Tool and the existing add section workflow are reverted.
There was a 45% decrease (-1.8 percentage points: 4.05% → 2.22%) in the percent of Junior Contributors blocked after posting a new topic using the New Topic Tool.
There was an 11.6% decrease (-1.3 percentage points: 11.2% → 9.9%) in the percent of new topics reverted 48 hours after they were posted using the New Topic Tool; however, there is not sufficient evidence to confirm this observed change was due to the New Topic Tool.
Senior Contributors are more successful publishing topics using the existing add section workflow than they are using the New Topic Tool.
66% of Senior Contributors that opened the New Topic Tool were able to successfully publish at least one new topic during the A/B test compared to 71% of Senior Contributors using the existing add new section workflow (- 5 percentage points; 7% decrease).
The New Topic Tool led to a slightincrease in the new topic completion rate for logged-out contributors; however, there is not sufficient evidence to confirm this observed change was due to the New Topic Tool.
9.6% of logged-out contributors that opened the New Topic Tool were able to successfully publish at least one new topic during the A/B test compared to 8.1% of Junior Contributors using the existing add new section workflow (+ 1.5 percentage points; 18.5% increase).
The New Topic Tool led to slight increases in the revert rate for new topic tool edits made by logged-out users and logged-in Senior Contributors; however, there is not sufficient evidence to confirm either of these observed changes were due to the New Topic Tool.
There was a 0.6 percentage point increase (1.5% → 2.1%; 40% increase) in the revert rate for new topic tool edits made by Senior Contributors compared to edits they made using the previous add section link.
There was a 3 percentage point increase (20.7% → 23.7%; 14.5% increase) in the revert rate for new topic tool edits made by logged-out users compared to edits they made using the previous add section link.
In this first analysis, we sought to learn whether people who have the New Topic Tool enabled as a beta feature are finding the tool disruptive and behaving in the ways they expect. We were also curious to begin to understand how heavily people were depending on the tool to start talk page discussions.
To determine the extent to which people were finding the tool "disruptive" and "behaving in ways they expect," we looked at the following metrics:
The percent of distinct contributors that explicitly disabled the New Topic Tool after publishing at least one new topic with the tool.
The percent of all published edits made with the New Topic Tool that are reverted within 48 hours of being published compared to the percent of all talk page edits made with full-page editing that are reverted within 48 hours of being published.
To begin to understand how "heavily people were depending on the tool to start talk page discussions", we looked at the following metrics:
The percent of distinct contributors who published at least one new topic with the tool.
For contributors that have published at least 1 new topic with the New Topic Tool, the percent of distinct contributors that used the New Topic Tool to create the following percentage of all new topics: 0-25%, 26-50%, 51-75%, or 76-100% of new topics.
Timing
This analysis was completed on 17 August 2021.
Conclusions and Findings
Conclusion: The New Topic Tool is not disruptive to the majority of people who have used it as well as to the people who maintain the pages on which the tool is being used.
Finding: 91.68% of contributors that published at least one new topic with the New Topic Tool kept the tool enabled during the reviewed time period.
Finding: Of all the edits Junior Contributors made to talk pages during the time period that was analyzed, the proportion of revertededits Junior Contributors made with the New Topic Tool was 21.3% less than the proportion of reverted edits they made with the existing, full-page editing interface.
Conclusion: Among people who started multiple discussions during the analysis period, nearly all of them came to depend on the New Topic Tool to start new topics.
Finding: Of the contributors that published more than one new topic on a talk page including at least one with the New Topic Tool, 95.3% of these contributors published between 75 to 100 percent of all their new topics using the New Topic Tool.
Percent of Junior Contributors that completed at least one comment attempt on a talk page during the AB test. | SourceThe percent of comments made by Junior Contributors on talk pages that were reverted within 48 hours of being published. | Source
In this last analysis, we sought to learn two things:
Does the Reply Tool cause Junior Contributors to be more successful publishing comments on talk pages?
Does the Reply Tool cause more disruption to other volunteers?
To answer the questions above, we ran an A/B test of the Reply Tool from 11 February through 10 March on 22 Wikipedias. The results from this test can be found in the "Findings" section below.
Timing
This analysis was completed on 23-April-2021. The A/B test ran from 11-Febraury-2021 through 10-March-2021.
Conclusions
The clear evidence that shows the Reply Tool causes a greater percentage of Junior Contributors to publish a comment without a significant increase in disruption, signals to the team that it would benefit all Wikimedia Projects and volunteers to have the Reply Tool made available as an opt-out preference.
Findings
The data below included logged-in users who had not previously interacted with the Reply Tool (defined as users whose discussiontools-editmode preference was empty).
72.9% of Junior Contributors who had access to the Reply Tool were able to publish at least one comment during the A/B test compared to 27.6% of Junior Contributors who did NOT have access to the Reply Tool.
Junior contributors were 7.2 times more likely to publish a comment than junior contributors using page editing.
There was a 79.5% decrease in the revert rate of comments Junior Contributors made with the Reply Tool compared to comments Junior Contributors made with page editing.
The next analysis we did was needed to help determine whether people, across experience levels, were having success using the Reply Tool and whether people were using the tool in ways that degrades the experiences of others.
This analysis's findings were used to decide whether the Reply Tool is functioning well enough for its impact on user behavior to be tested via a larger-scale (read: at wikis beyond our partner wikis) A/B test.
Timing
This analysis was completed on 5-January-2021
Conclusions
The high rate at which people who used the Reply Tool to publish the comments they started writing, combined with the lack of clear evidence of disruption, led the team to think the Reply Tool was ready to be tested at a larger scale via an A/B test. More details below.
Findings
The edit completion rate for Junior Contributors using the Reply Tool is 4x higher than the edit completion rate for people using full-page wikitext editing. Source
People who have made <1,000 cumulative edits, are between 32% and 510% more likely to publish edits with the Reply Tool (across modes) than they are with full page wikitext talk page editing. Source
<1% of comments Junior Contributors post with the Reply Tool are reverted within 48 hours, across wikis (this number is higher for Senior Contributors ). Source
In this first analysis, we sought to learn whether people at the Arabic, Dutch, French and Hungarian Wikipedias, where the Reply Tool had been available as a Beta Feature since 31-March-2020, were finding the tool valuable.
To determine the extent to which people were "finding the tool valuable" we looked at how frequently people were using the tool (as measured by the number of distinct days they use it on) and how intensely people were using the tool (as measured by the total number of edits they make with the Reply Tool and the percentage of total talk page edits they used the Reply Tool to make).
Understanding the above helped us determine the tool was ready to be made available as an opt-out feature at these wikis and made available as an opt-in Beta Feature at others.
Timing
This analysis was completed on 28-July-2020.
Findings
The data below included people who have used the Reply Tool as a Beta Feature at the Arabic, Dutch, French and Hungarian Wikipedias, between 31-March and 30-June-2020:
328 people posted at least one comment using the Reply Tool
69.5% of people posted ≥ 2 comments with the Reply Tool
61.3% of people posted a comment using the Reply Tool on ≥ 2 distinct days[1]
23.6% of people used the Reply Tool to make over half of their talk page edits[2]
↑
A "distinct day" was defined as a distinct calendar day. Therefore, some of these edits may have occurred 24 hours apart and some may have occurred only a few hours apart depending on what time of day the edit was made.
↑
This data is a bit noisy in so far as there are some edits that cannot be made with the Reply Tool (e.g. starting a new section, editing existing content, etc.) and the software is not currently able to differentiate between comments and other talk page edits.
,_spring_2023.png)
_spring_2023.png)
