Below you can read more about what this experiment demonstrated, what the Editing Team is planning in response, and more details about the test's design.
Conclusion and next step
Reference Check caused an increase in the quality of edits newcomers publish and did not cause any significant disruption.
This combination is leading the Editing Team to be confident that offering Reference Check as a default-on feature would have a net positive impact on wikis and the people who contribute to them.
Findings
There was 2x increase in the proportion of new content edits by newcomers, Junior contributors, and unregistered users that included a reference when Reference Check was shown to eligible edits.
New content edits *with* a reference
People shown the Reference Check are 2.2 times more likely to publish a new content edit that includes a reference and is constructive (not reverted within 48 hours).
Increases were observed across all reviewed user types, wikis, and platforms.
The highest observed increase was on mobile where contributors are 4.2 times more likely to publish a constructive new content edit with a reference when Reference Check was shown to eligible edits.
Revert rate
New content edit revert rate decreased by 8.6% if Reference Check was available.
New content edits by contributors from Sub-Saharan Africa are 53% less likely to be reverted when Reference Check is shown to eligible edits.
We observed increases on both desktop and mobile. On mobile, users are 4.2 times more likely to include a reference with their new content when the reference check is shown to eligible edits.While some non-constructive new content edits with a reference were introduced by this feature (5 percentage point increase), there was a higher proportion of constructive new content edits with a reference added (23.4 percentage point increase). As a result, we observed an overall increase in the quality of new content edits.
There was a -8.6% decrease in the revert rate of all new content edits comparing edits where Reference Check was shown in the test group to edits that were eligible but not shown Reference Check in the control group.
Constructive Retention Rate
Contributors that are shown Reference Check and successfully save a non-reverted edit are 16 percent more likely to return to make a non-reverted edit in their second month (31-60 days after).
This increase was primarily observed for desktop edits. There was a non-statistically significant difference observed on mobile.
Guardrails
Edit Completion Rate
We observed no drastic decreases in edit completion rate from intent to save (where Reference Check is shown) to save success overall or by wiki.
Overall, there was a 10% decrease in edit completion rate for edits where Reference Check was shown.
There was a higher observed decrease in edit completion rate on mobile compared to desktop. On mobile, edit completion rate decreased by -24.3% while on desktop it decreased by -3.1%.
Block Rate
There were decreases or no changes in the rate of users blocked after after being shown Reference Check and publishing an edit compared to users in the control group.
False Negative Rate
There was a low false negative rate. Only 1.8% of all published new content edits in the test group did not include a new reference and were not shown Reference Check.
False Positive Rate
6.6% of contributors dismissed adding a citation because they indicated the new content being added does not need a reference. This was the least selected decline option overall.
Users in the test group were shown the Reference Check notice prompting them to decide whether the new content they were adding need a reference (if they had not already added one themselves).
User in the control group were shown the default editing experience, even if they did not accompany the new content they were adding with a reference.
Timing
This analysis was completed on 16 April 2024 and analyzed engagement data at the 11 participating wikis from 18 February 2024 through 4 April 2024.
