Wikimedia Apps/Team/Android/Recommended Content in Search

As part of the Android team’s 2024/2025 Annual Plan, the Android team will explore various ways to increase reader retention through browsing and learning experiences. This project page documents the Android team’s experimentation related to the Wikimedia Foundation 2024-2025 Annual Plan, specifically the Wiki Experiences 3.1 Key Result.

Current Status

2024-08-21: sharing hypothesis and designs.
Next: community consultation and in-app feedback to pilot wikis and regions.

Summary

Research revealed the search is the most reportedly visited part of the app. Our quantitative data aligns with the self reflections of our users through surveys. We have also learned that there is an interest from app users to receive content recommendations.

The Research team conducted a study about Non-Editing Participation, which reinforced an opportunity across platforms to suggest content in search based on user interest. Namely, the Research team found that roughly 50% of survey respondents find articles of interest via internal links on Wikipedia or by starting on the mainpage and then going to Search. These insights have contributed to a hypothesis of suggesting articles in the Wikipedia Search.

Background

How does this work fit into the Wikimedia Foundation's Annual Plan?

Wiki Experiences 3: Consumer experience (Reading & Media)

Under the Wikimedia Foundation's Infrastructure Goal, and within the group of objectives focused on Wiki Experiences, is an objective related to improving the experience of consumers:

Wiki Experiences 3: Contributor experience Objective: A new generation of consumers arrives at Wikipedia to discover a preferred destination for discovering, engaging, and building a lasting connection with encyclopedic content.

Wiki Experiences 3.1 Key Result

Under the consumer experience objective, is one key result focused on increasing reader retention:

Wiki Experiences 3.1 (WE3.1) Key Result: Release two curated, accessible, and community-driven browsing and learning experiences to representative wikis, with the goal of increasing the logged-out reader retention of experience users by 5%.

Several Wikimedia Foundation teams are committed to working on projects under the WE3.1 Key Result: Draft_Hypotheses.


Android team hypothesis	Timeline	Phabricator epic
Wiki Experiences 1.2.3: If we enhance the search field in the Android app to recommend personalized content based on a user's interest and display better results, we will learn if this improves user engagement by observing whether it increases the impression and click-through rate (CTR) of search results by 5% in the experimental group compared to the control group over a 30-day A/B/C test. This improvement could potentially lead to a 1% increase in the retention of logged out users.	July 1, 2024 - September 30, 2024	T370117

Wikimedia Foundation teams are approaching annual planning more iteratively this year, so rather than committing to larger year-long projects, our first hypothesis is fairly narrow in scope. This should allow us to deliver value in smaller increments throughout the year, while also ensuring we have the flexibility to pivot as we learn. So WE1.2.3 is just a first step, future Growth team work will likely relate to implementing what we learned from WE1.2.3 user tests, and possibly developing new types of structured tasks later in the fiscal year.

Community Discussion

We have discussed the broader concept behind this project with communities as part of the WMF annual plan Product & Technology OKRs discussion, but we will also initiate a more detailed community consultation with our pilot wikis and regions:

South Asian readers on Hindi and English Wikipedia.
Sub Saharan Africa readers on English, French, and Arabic Wikipedia.

Design

Test B General
Test C Personalized

Upon tapping into recommended article, users in tests B and C will both be served a survey after spending 10 seconds in the article:

Initial Screen
Upon tapping into Feedback box
Survey Confirmation

Measurement and Results

How will we know we were successful

The team is planning to run a 20 days ABC test in the app to evaluate our hypothesis, along with running a survey to understand user satisfaction with search recommendations. The control group in the experiment will see our current search screen. Experiment Group B will see search recommendations using the Top read and In The News APIs. Experiment Group C will see search recommendations using the Because you read and Nearby APIs for a more personalized recommendations experience. We will validate our hypothesis for the experiment groups with the key indicators and guardrails below.

Validation:

Search Satisfaction rate of 65%.
1% higher search retention rate from experiment group vs control during experiment period.
5% of unique users click suggestion in search more than once in a 20 days period.
5% increase in CTR of Search from experiment group compared to control group.
Personalized suggestions has 10% higher CTR than Generalized Suggestions.

Guardrails:

Experiment group doesn't have a higher abandonment rate than control.
No more than 2% of feedback includes reports of NSFW, Vandalism or Offensive recommendations.
Search doesn't worsen geographic bias.

Curiosities:

Required: Do we see a difference in metrics between logged in and logged out users?
Does the preference for the type of content shown in search differ by platform and language?
Would users like to see suggestions presented somewhere other than search?
Do people return to the search just to click a suggestion?
Should there be a filter for the type of content suggested (BLP, NSFW, Controversial Topics, etc.?)

After conducting data analysis we will share the results of the experiment on this page, and in partnership with the community determine if there should be further investment and rerelease of content suggestions or if we should scale another hypothesis instead.

Updates

Weekly updates can be found on our main updates page.