Between July and December 2024, the Web team conducted experiments checking how to make it easier for readers to discover information on the wikis. This was part of our annual plan focused on content discovery for readers.
|
تجارب اكتشاف المحتوى
|
The team ran four experiments. This page outlines the details on each experiment, as well as our findings and timeline.
The results of the experiments will be used to identify which features we will build for the future, based on how useful they are proven to be in helping readers find useful and interesting information across the wikis.
التحديثات
الخلفية
As new generations of readers come to our sites, we want to make it easy for them to learn. This includes making it easier to find the information they need across articles and pages as well as making it easier to discover new information based on their needs or interests. To do this, we hope to leverage advances in technology and build out new capabilities within our platform that will allow us to make discovery and browsing easier than it was in the past. This goal led the team to focus on more experimental work to begin with, as a way of being able to test out more features quickly. We started with the following hypothesis:
We designed and qualitatively evaluated three proofs of concept focused on building curated, personalized, and community-driven browsing and learning experiences. This allowed us to estimate the potential for increased reader retention (experiment 1: providing recommended content in search and article contexts, experiment 2: summarizing and simplifying article content, experiment 3: making multitasking easier on wikis)
This hypothesis focuses on identifying ideas for features or projects that would make it easier to browse and learn across the wikis on the desktop and mobile websites. Work here included identifying and building proofs of concepts for each idea and providing the results of initial tests on the idea.
The goal is to find ideas that we think would work well at a larger scale and commit to building them and providing them to the wikis.
This work begins discovery into areas that we have not yet been working on in the past, such as summarizing or remixing content. Working across communities to ensure proper editing and moderation workflows for these new content types will be crucial.
Experiment 1: Display article recommendations in more prominent locations, search
This experiment sought to add suggestions to the empty state of our search bar, providing readers with recommendations on pages they could read next. We wanted to experiment with whether readers click on recommendations in the empty state of the search bar to see if this type of feature is helpful to readers.
In order to perform this experiment, we built a browser extension which injects the proposed recommendations into the search bar. Then we tracked clicks on the provided suggestions. We also considered using the extension itself to communicate to readers when the experiment is over.
Experiment hypothesis
If we display recommendations in search, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to existing recommendation entry points (Related pages).
Experimental setup
A quicksurvey was displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to download a browser extension. Interested readers were prompted to download a browser extension which added the feature into the page.
Data was collected on how often readers clicked on the suggestions generated by the extension. No user or session data was collected.
النتائج
We found that the readers interacted with the recommendations shown in the search bar at a rate higher than recommendations shown at the bottom of the page.
Report on the overall clickthrough rate (CTR) for search recommendations coming from the browser extension:
- On English wikipedia (enwiki) desktop, the overall CTR for search recommendation is 1.53%
- On Spanish wikipedia (eswiki) desktop, the overall CTR for search recommendation is 0.72%
- The overall CTR across both wikis is 1.44%
Comparison to clickthrough rate for related pages on desktop, mobile:
We observed that the CTR for search recommendations is higher than that for related pages on desktop for both English Wikipedia and Spanish Wikipedia
- English Wikipedia (enwiki): 1.53% vs 0.41% (CTR of related pages on desktop, enwikivoyage), diff 1.12 pp
- Spanish Wikipedia (eswiki): 0.72% vs 0.13% (CTR of related pages on desktop, eswikivoyage), diff 0.59 pp
Detailed results are available on T374965.
Next steps
On mobile:
- Proceed with A/B test for the feature on one or more pilot wikis (pilot wiki list to be selected). Details on A/B test setup are available in phab:T378094
On desktop:
- Wait until the results of the article-level recommendations are ready and compare and proceed with the feature which shows better results.
Experiment 2: Improve article recommendations APIs
We wanted to compare various existing and proposed APIs recommending content independent of the user interface. This work allowed us to test different recommendation methods against one another to determine which ones are the most useful for readers.
Experiment hypothesis
If we compare readers preferences for suggested article APIs, we will learn which API readers find most useful.
Experimental setup
A quicksurvey was displayed to a small percentage of readers on English and Spanish Wikipedias, inviting them to take a survey. The survey presented recommendations from three different APIs. Users were be asked to submit which recommendations were most useful and interesting to them.
Experiment 3: Display article recommendations in more prominent locations, article pages
We wanted to explore displaying recommendations in other parts of the page. This experiment focused on displaying recommendations alongside article content on desktop.
Experimental setup
The experiment code was loaded in the Wikipedia Recommendations browser extension. Users of the browser extension saw the recommendations immediately.
Experiment hypothesis
If we display recommendations alongside article pages, people will be interested in reading the recommended pages, as shown through a clickthrough rate similar to, or higher than, existing recommendation entry points (Related pages, Experiment 1).
Results
- On English wikipedia (enwiki) desktop, the overall CTR for article recommendations is 0.54%
- On Spanish wikipedia (eswiki) desktop, the overall CTR for article recommendations is 0.28%
We observed that the CTR for sidebar article recommendations is higher than that for related pages on desktop (listed at the bottom of articles) for both English Wikipedia and Spanish Wikipedia
- English Wikipedia (enwiki): 0.41% ->0.54% , increased 0.13 pp
- Spanish Wikipedia (eswiki): 0.13% -> 0.28%, increased 0.15 pp
Hypothesis was successful:
✅ Readers will want to click on recommendations on the right of the article (measured by comparing the CTR of recommendations in search with other recommendation locations - specifically related pages on desktop).
Next steps
Sine the hypothesis was successful, we recommend to proceed with an A/B test of the feature on the desktop website.
Experiment 4: Article Summaries
Research
- Wikimania 2024, "Written by AI" How do editors and machines collaborate to create content. At Wikimania 2024, Wikimedians discussed ways that AI/machine-generated remixing of our existing content can be used to make Wikipedia more accessible and easier to learn from. The goal for the new features is to help a new generation of readers discover and learn from reliable, encyclopedic information on Wikipedia. The session focused on ways to give control and oversight to editors and communities over the structure and content of these new concept and feature ideas.
- Usability study on Simple Summaries. We ran an unmoderated Userlytics study on a prototype of our Simple Summary feature with 8 participants. The prototype included a summary of the introduction of the English Wikipedia article on Dopamine. We analyzed videos and screen recordings of the sessions. Participants found simple summaries easy to use, useful, and had an appropriate level of trust in the machine-generated summary. Responses were much more positive than expected. The main issue to be addressed before release is ambiguity between the Simple Summary and the main article content while the simple summary accordion is closed.
الجدول الزمني
| Estimated time | Planned activity |
|---|---|
| |
| Building experiments | |
| |
| Running experiment 4 |