Reference Check demo at Wikimania (2024) by Selena Deckelmann, Wikimedia Foundation's Chief Product and Technology Officer
T341308 - Refining the user experience for showing multiple Edit Checks within a single edit
T360591 - Developing an API to increase the speed and ease with which new Edit Checks and Suggested Edits can be introduced within/alongside the visual editor.
T364505 - Creating a UI framework to ensure people experience new Edit Checks and Suggested Edits as part of a common language
T368274 - Researching the feasibility of using a large language model to detect peacock behavior within the edits people are making.
Looking ahead, the Editing Team will continue developing Edit Check during the 2024-2025 fiscal year. This work will support the Wiki Experiences Objective.
Note: if you are interested in enabling the Reference Check at your wiki, please ping User:Trizek (WMF) or User:PPelberg (WMF), or contact us on our talk page.
Wednesday's conversation will be focused on the following Edit Check-related topics:
Discussing the user experience for Paste Check. This next Edit Check we'll be introducing is meant to help newcomers adding new content unknowingly make a copyright violation.
Reviewing the state of the Edit Check proejct. What Checks are available at what wikis? What new Check are being planned? More broadly, what strategy is guiding the Edit Check in this coming year? Etc.
On Tuesday (18 June), Reference Check became available by default at all Wikipedias except bn, de, en, hi, id, nl, pl, and ru.[3]
Feedback people will receive when attempting to link to a blocked domain within the visual editor.
Beginning Thursday (13 June), people who attempt to add an external link in the visual editor (desktop and mobile) will receive immediate feedback when they attempt to link to a domain a project has decided to block.
Link Check (as we are calling it) evaluates all external domains people attempt to link to against the following sources:
Local lists:MediaWiki:BlockedExternalDomains.json and MediaWiki:Spam-blacklist
Global list:meta:Spam_blacklist
Technical demo showing what it could be like for Edit Checks to appear within/alongside the editable surface.
While presenting Checks in the moments above have proven effective, for a while, volunteers and staff have been sharing ideas for Checks that could appear while people are actively adding/changing content within the article.
With this initialtechnical demo, the Editing Team is exploring what it could be like for Checks to:
Appear within/alongside the editable document, in the moment when people make a change that causes a check to be activated
Respond when people make a change that impacts the Check that they activated
What do you think about this idea? What questions/concerns/ideas/etc. does it bring to mind?
On 7 March, the first iteration Reference Reliability check became available to everyone at all wikis, by default.
This means that whenever anyone attempts to cite a source that a project has blocked, they will be made aware directly within Citoid and prompted to try another source.
Where "blocked" on in this context means the domain someone is attempting to cite is lited on MediaWiki:Spam-blacklist or MediaWiki:BlockedExternalDomains.json.
In the meantime, you can see the kinds of edits EditCheck currently thinks warrant a reference, by filtering Recent changes using the newly-introduced editcheck-references tag. View the tag on en.wiki and fr.wiki.
Informed by community conversations (still ongoing)[4][5][6][7] and a series of technical and design investigations[8][9][10], during February the Editing Team became clear about the first version of Edit Check on mobile...
編集チェック機能の初期版でシステムが脚注の追加をうながすと、利用者はこんな体験をした。
User Experience: the first version of Edit Check will introduce a new step within the mobile visual editor's publishing workflow that people will see if/when they add new content without a reference. Design for the desktop user experience is still underway. See T329579.
Usability Testing: to learn whether people understand and can intuitively navigate the mobile Edit Check workflow, we will soon begin a series of usability tests. See T327356.
Technical Investigation: Edit Check will use a "transaction-based" approach for determining what new content is added within a given edit session. Work on developing a way to detect individual sentences is ongoing in T324363.
Initial Heuristic: To start, the initial Edit Check heuristic will be relatively straightforward in so far as it will prompt people to decide whether the change they are making warrants a reference if/when they are adding a new paragraph and that paragraph does not already contain a reference. See T324730 and T329988#8654867.
Next up: the Editing Team will be implementing the initial Edit Check heuristic (T324730) and a corresponding hiddenchange tag (T324733) so that we – volunteers and members of the Editing Team – can evaluate the extent to which the reference check heuristic is getting initiated in expected cases.
Work on Edit Check is underway! Below you will find an overview of what the Editing Team is actively working on…
Community conversations: Between October 2022 and January 2023, the Editing Team hosted seven community conversations to learn what contributing to Wikipedia has been like for people living in and from Sub-Saharan Africa. Next week, you can expect the team to publish the findings from these conversations and how they will inform the work we do on this project.
Initial Focus: The first feature the team will be introducing is one that checks whether the new content people are attempting to add includes a reference. Learn more in the Strategy and Approach section below.
Design: The team is actively working on a proposal for what the mobile user experience for the first reference check could be like. In the coming weeks, we will be inviting volunteers to help us revise and refine these designs. In the meantime, you can follow along with this work in Phabricator.
Talking with experienced volunteers: for the "reference check" to be useful to inexperienced and experienced volunteers alike, it will need to guide people to cite references in ways that projects expect. In the coming weeks, we'll begin conversations with experienced volunteers to learn what these expectations are so that we can ensure Edit Check is configured in ways that align with them.
Technical investigations: For the "reference check" to work, the software will need to know when people are attempting to add new content, whether that new content warrants a reference, and whether it currently contains a reference. The Editing Engineering team is currently doing a series of technical investigations to decide how we will approach building this functionality.
戦略とアプローチ
To equip newcomers and Junior Contributors from Sub-Saharan Africa with the know-how and tools to publish changes they are proud of and that experienced volunteers consider useful, the Editing Team will be introducing new functionality within the visual editor (desktop and mobile) that will check the changes people are attempting to make and present them with actions they can take to improve these changes in ways that will align with established Wikipedia policies and guidelines.
The first "check" the Editing Team will be introducing is one that will detect when people are attempting to add new content to an existing article without a corresponding reference and prompt them to do so. The functionality will be accompanied by a complimentary set of features that will enable moderators to configure the user experience newcomers and Junior Contributors will see to ensure the software is guiding them to take actions that align with project policies and conventions.
問題点
The visual editor's growing popularity among people who are new to editing Wikipedia[11] leads us to think that the editing experience has been reasonably successful at helping inexperienced volunteers learn the technical skills necessary to publish changes to Wikipedia.
The trouble is that the visual editor and other editing interfaces do not make people aware of the Wikipedia policies and guidelines they are expected to follow.
As a result, the changes inexperienced volunteers publish often break established best practices and lead to undesirable outcomes for inexperienced volunteers, experienced volunteers, and Wikipedia projects as a whole:
Inexperienced volunteers become disappointed and frustrated when the good-faith change(s) they arrived to the wiki seeking to make are undone (read: reverted), deleted, and/or scrutinized in inequitable ways. These poor interactions are demotivating and drive these could-be volunteers and community members, and the knowledge that are uniquely positioned to offer, away.[12]
Experienced volunteers/moderators need to do more work reverting low-quality edits and posting messages on inexperienced volunteers' talk pages to make them aware of the policies and/or guidelines they are likely to have unknowingly broken. Continually needing to educate inexperienced volunteers and undo their changes can lead to experienced volunteers becoming skeptical of inexperienced volunteers and impatient with them.
Wikipedia projects struggle to grow and diversify their volunteer populations and shrink the knowledge gaps present within Wikimedia wikis.
This project seeks to address the challenges above by:
Offering inexperienced volunteers relevant and actionable feedback about Wikipedia policies in the precious moments when they are in the midst of making a change using the visual editor.
Equipping moderators with a new ability to specify the feedback inexperienced volunteers are presented with while they are editing
変化の理論
This project is built on the belief that by surfacing relevant guidance in the precious moments when inexperienced volunteers are in the midst of making a change to Wikipedia and equipping them with the know-how and tools necessary to apply this guidance, they will make changes they are proud of and that experienced volunteers value.
In the longer term, the Editing Team thinks that people who are new, particularly people who have historically been excluded from and harmed by established power structures, will feel safe and motivated making changes to Wikipedia if they can accurately predict whether the changes they are attempting to make are aligned with existing Wikipedia policies, guidelines, and/or cultural conventions.
More broadly, the Editing Team thinks that to evolve towards a future where wikis' policies and cultural norms – and ultimately, content – reflect the diverse experiences of the people these projects are intended to serve, we first need to make the norms and standards that are currently in place legible and actionable to people while they are editing.[13] This way, volunteers can develop shared awareness of cases where these norms and standards are not having the impacts they were intended to have and decide what – if any – changes they think are worth making to them in response.
Below you can read more about what this experiment demonstrated, what the Editing Team is planning in response, and more details about the test's design.
Conclusion and next step(s)
Reference Check caused an increase in the quality of edits newcomers publish and did not cause any significant disruption.
There was 2x increase in the proportion of new content edits by newcomers, Junior contributors, and unregistered users that included a reference when Reference Check was shown to eligible edits.
New content edits *with* a reference
People shown the Reference Check are 2.2 times more likely to publish a new content edit that includes a reference and is constructive (not reverted within 48 hours).
Increases were observed across all reviewed user types, wikis, and platforms.
The highest observed increase was on mobile where contributors are 4.2 times more likely to publish a constructive new content edit with a reference when Reference Check was shown to eligible edits.
Revert rate
New content edit revert rate decreased by 8.6% if Reference Check was available.
New content edits by contributors from Sub-Saharan Africa are 53% less likely to be reverted when Reference Check is shown to eligible edits.
We observed increases on both desktop and mobile. On mobile, users are 4.2 times more likely to include a reference with their new content when the reference check is shown to eligible edits.While some non-constructive new content edits with a reference were introduced by this feature (5 percentage point increase), there was a higher proportion of constructive new content edits with a reference added (23.4 percentage point increase). As a result, we observed an overall increase in the quality of new content edits.
There was a -8.6% decrease in the revert rate of all new content edits comparing edits where Reference Check was shown in the test group to edits that were eligible but not shown Reference Check in the control group.
Constructive Retention Rate
Contributors that are shown Reference Check and successfully save a non-reverted edit are 16 percent more likely to return to make a non-reverted edit in their second month (31-60 days after).
This increase was primarily observed for desktop edits. There was a non-statistically significant difference observed on mobile.
Guardrails
Edit Completion Rate
We observed no drastic decreases in edit completion rate from intent to save (where Reference Check is shown) to save success overall or by wiki.
Overall, there was a 10% decrease in edit completion rate for edits where Reference Check was shown.
There was a higher observed decrease in edit completion rate on mobile compared to desktop. On mobile, edit completion rate decreased by -24.3% while on desktop it decreased by -3.1%.
Block Rate
There were decreases or no changes in the rate of users blocked after after being shown Reference Check and publishing an edit compared to users in the control group.
False Negative Rate
There was a low false negative rate. Only 1.8% of all published new content edits in the test group did not include a new reference and were not shown Reference Check.
False Positive Rate
6.6% of contributors dismissed adding a citation because they indicated the new content being added does not need a reference. This was the least selected decline option overall.
Users in the test group were shown the Reference Check notice prompting them to decide whether the new content they were adding need a reference (if they had not already added one themselves).
User in the control group were shown the default editing experience, even if they did not accompany the new content they were adding with a reference.
Timing
This analysis was completed on 16 April 2024 and analyzed engagement data at the 11 participating wikis from 18 February 2024 through 4 April 2024.
Evaluating impact
The viability of the features introduced as part of the Edit Check project depends on the impacts it causes and averts.[19]
This section describes the:
Impacts the features introduced as part of the Edit Check are intended to cause and avert
Data we will use to help[20] determine the extent to which a feature has/has not caused a particular impact
Evaluation methods we will use to gather the data necessary to determine the impact of a given feature
Increase the likelihood that newcomers and Junior Contributors editing from within Sub-Saharan Africa will accompany the new content they are adding with a reference
Increase in the percentage of published edits that add new content and include a reference
Increase in the percent of newcomers or Junior Contributors from SSA that publish at least one new content edit that includes a reference
Increase in the likelihood that someone includes a reference the next time they contribute new content.
Newcomers and Junior Contributors editing from within Sub-Saharan Africa will report feeling safe and confident making changes to Wikipedia
Newcomers and Junior Contributors find the feedback and calls to action Edit Check presents them with to be:
Helpful
Supportive
Motivating
Qualitative feedback via channels like:Community Calls, talk pages, event organizers, etc.
4.
Experienced volunteers will independently audit and iterate upon Edit Check's default configurations to ensure Edit Check is causing newcomers and Junior Contributors to make productive edits.
5.
Newcomers and Junior Contributors will be more aware of the need to add a reference when contributing new content because the visual editor will prompt them to do so in cases where they have not done so themselves.
Increase in the percent of newcomers or Junior Contributors from SSA that publish at least one new content edit that includes a reference.
Proportion of contributors that are presented Edit Check feedback and abandon their edits (indicated by event.action = abort and event.abort_type = abandon)
Proportion of contributors blocked after publishing an edit where Edit Check was shown is significantly higher than edits in which Edit Check was not shown
Proportion of new content edits published without a reference and without being shown Edit Check (indicator of false negative)
Proportion of contributors that dismiss adding a citation and select "I didn't add new information" or other indicator that the change they are making doesn't require a citation
A/B test[23], qualitative feedback received from volunteers about the accuracy and usefulness of Edit Check's current configuration [25]
7.
Edit Check is too resource intensive to scale
Efficiencies do not emerge over time making each new Edit Check as "expensive" to implement as the first one
↑
Where "viability" in this context refers to a feature being fit for being scaled to all projects as determined by the extent to which it has been proven to have a net positive impact on wikis and the volunteers who build and maintain them.
↑Emphasis on "help" seeing as how all decisions will depend on a variety of data, all of which need to be weighted and considered to make informed decisions.
↑T325838: Finish Edit Check measurement plan proposal
↑ 22.022.122.2At every project where Edit Check is available, volunteers will be able to use the editcheck-reference-activated tag to review edits where the reference check is shown to people in the process of publishing an edit. Learn more about Edit Check tags.
↑T325851: Conduct pre-mortem for Edit Check project
↑In addition to existing feedback channels (Phabricator, talk pages, etc.) there will be a minimum of two additional ways for people to share feedback about Edit Check: A) reporting edits that you think Edit Check should not have been shown within and B) declining to add a reference mid-edit by indicating you think Edit Check was shown when it shouldn't have been.
.png)
.png)
Reference Reliability (mobile)
Reference Reliability (desktop)
導入時のモバイル版編集チェック機能の体験は、利用者に出典をつけるかどうか判断を促す。
編集チェックにオススメされ、思い切って出典を付けてみたら、好意的なフィードバックをもらえたとします。
出典追加をしたくない ボタンを選んだ人に、理由を書いてもらうよう誘導する画面（スクリーンショット）
モバイル版の編集チェック機能の概要を、モバイル版の体験を示すいくつかの画面で紹介。
