Readers/2024 Reader and Donor Experiences/Content Discovery

The content discovery project is documented in the 2024-2025 Annual Plan for the Wikimedia Foundation, as a Key Result WE3.1 for the Product and Technology department. It is focused on improving the ways readers discover content across Wikipedia. The goal is to make it easier to access and learn from knowledge on the wikis, which will be measured through the retention of readers (how often readers come back to Wikipedia).

Key Result

Release two curated, accessible, and community-driven browsing and learning experiences to representative wikis, with the goal of increasing the logged-out reader retention of experience users by 5%.

Background

As new generations of readers come to our sites, we want to make it easy for them to learn. This includes making it easier to find the information they need across articles and pages as well as making it easier to discover new information based on their needs or interests. To do this, we hope to leverage advances in technology and build out new capabilities within our platform that will allow us to make discovery and browsing easier than it was in the past.

This KR focuses on increasing the retention of a new generation of readers on our website through making it easier to discover and browse Wikipedia content. The KR will include various explorations and the development of new curated, personalized, and community-driven browsing and learning experiences (for example, feeds of relevant content, topical content recommendations and suggestions, community-curated content exploration opportunities, etc.).

We plan on beginning the fiscal year by experimenting with a series of experiments of browsing experiences to determine which we would like to scale for production use, and on which platform (web, apps, or both). We will then focus on scaling these experiments and testing their efficacy in increasing retention in production environments. Our goal by the end of the year is to launch at least two experiences on representative wikis and to accurately measure a 5% increase in reader retention for readers engaged in these experiences.

Measurement Plan

To be effective at achieving this KR, we will require the ability to A/B test with logged-out users, as well as instrumentation capable of measuring reader retention. We might also need new APIs or services necessary to present recommendations and other curation mechanisms. We are currently committed to providing these capabilities, but since they carry some risk of evolving complexity, we have also identified alternative metrics and measurement plans that we can use in the case that A/B testing and retention prove impossible to achieve within the current year across platforms.

Hypotheses

Experimenting with browsing and learning experiences (hypothesis 3.1.1)

Designing and qualitatively evaluating three proofs of concept focused on building curated, personalized, and community-driven browsing and learning experiences will allow us to estimate the potential for increased reader retention (experiment 1: providing recommended content in search and article contexts, experiment 2: summarizing and simplifying article content, experiment 3: making multitasking easier on wikis)

Hypothesis background

This hypothesis focuses on identifying ideas for features or projects that would make it easier to browse and learn across the wikis on the desktop and mobile websites. Work here will include identifying and building proofs of concepts for each idea and providing the results of initial tests on the idea. The goal here would be to find ideas that we think would work well at a larger scale and commit to building them and providing them to the wikis.

It’s important to note that this work begins discovery into areas that we have not yet been working on in the past such as summarizing or remixing content. Working across communities to ensure proper editing and moderation workflows for these new content types will be crucial. We expect to collaborate heavily on this work with communities.

Making content easier to read (hypothesis 3.1.3)

If we develop models for remixing content such as a content simplification or summarization that can be hosted and served via our infrastructure (e.g. LiftWing), we will establish the technical direction for work focused on increasing reader retention through new content discovery features.

Hypothesis background

This hypothesis aims to make it easier to get a simple overview on a Wikipedia article. The hypothesis will also look into ways to automatically simplify the reading level of content so that it is more accessible to a wider audience of readers.

The results of these findings will be tested and evaluated and discussed with communities in the shape of one of the experiments in hypothesis 3.1.1.

Ensuring our we can provide recommendations for everyone (hypothesis WE 3.1.4 )

If we analyze the projected performance impact of hypothesis WE3.1.1 and WE3.1.2 on the Search API, we can scope and address performance and scalability issues before they negatively affect our users.

Hypothesis background

As part of the work in the KR, we want to explore adding recommendations in more prominent places throughout the site so that readers have opportunities to more easily discover content. To do this, we’ll need to review the APIs used for existing recommendation system - such as Related Pages or Suggested Edits and determine whether, and what, changes they would need in order to support the additional traffic expected to be generated from the more prominent location.

Search recommendations on the Android app (hypothesis WE 3.1.5)

If we enhance the search field in the Android app to recommend personalized content based on a user's interest and display better results, we will learn if this improves user engagement by observing whether it increases the impression and click-through rate (CTR) of search results by 5% in the experimental group compared to the control group over a 30-day A/B test. This improvement could potentially lead to a 1% increase in the retention of logged out users.

Hypothesis background

Users on the Android app have the ability to opt in to the collection of limited amounts of data such as the history of pages they have read. For users who have opted in, we have the ability to explore providing them with recommendations that are personalized based on their previous interests. This hypothesis explores the addition of these recommendations within the search bar.