Reading/Web/Projects/Related pages/Moving to stable

Data and performance

According to the timeline below, as of late October, related pages is now live on stable, on all Wikipedias, except Japanese, Spanish, English and French. According to our plan, it should be live on Japanese and Spanish, before discussing and moving to English and French.

Below are numbers, of feature performance so far:

Plan and analysis

The Reading team would like to move forward with making related pages a stable feature on mobile web on all Wikipedias (except for the German Wikipedia, which rejected the feature, unless they decided to re-think their vote). The decisions is based on the fact that feature performance on mobile after over 3 months of running in beta (+ 20% click-through rate) is enough matrix to enable us to move the feature to stable. The click through rate is an indication that readers are inclined to check further related topics on the articles as we offer them. This is a good starting point, in which further feature development can stem from.

Timeline and tasks

• Announce the plan and time line by Sept 1st
• Allow 3-4 weeks of discussion
• On Desktop: Remove Related pages from desktop beta feature on all Wikipedias --if a community needs an exception, let us know; we understand that the feature might be helpful in populating article in relatively small or new Wikipedias
• On mobile: Start a staged roll out, where we exclude the top 6 Wikipedias in terms of article count, deploy the feature to all other wikis at once. This helps making stable bug fixes easier, given smaller traffic
• If all goes well, start deployment according to the above plan by early October.

Community feedback summary

• The majority of feedback was concerned with the placement of inappropriate or redundant machine generated content on articles.
• Concerns/comments around how the algorithm behaves

Changes have been deployed to the way how the logic of the selected articles is implemented. Initially, 2/3 of the criteria for related pages were essentially popularity (links in and featured article status). Only 1/3 was similarity. We now moved it to being 100% about similarity.(Details)

• Concerns around the display non-free images on English WP

This has been resolved via Extension:PageImages fixes and will continue to have feature enhancement

• Concerns around the click-through rate as a sole metric for measuring feature success.

As mentioned above, the click through rate is a helpful indication to help us realize user interest in the idea of offering articles related to the topic they are reading/browsing.

• Related pages and See also sections

Currently, editors can override the suggested articles by placing articles manually. In the future we will be looking into different patterns of how see else is significant/present in each Wikipedia, and adapt Related Pages to be as beneficial as possible to each Wikipedia

Data

Summary

Clickthrough rates for related pages links are especially high (20.2%) when users see the related pages section at the bottom of the article. The overall clickthrough rate on related pages links for mobile is 7.5%. Rates are significantly (5.4 times) lower for desktop (1.2% and 2.8%, respectively). No significant changes in these rates were observed over time, indicating that users will continue using related pages with a similar frequency.

Details

On mobile, the clickthrough rate for any related link given is 0.0748 clicks per page (ranging between 0.050 and 0.099 with 95% confidence). When users view the related pages (by scrolling to the bottom of the article), the clickthrough rate is 0.2019 clicks per page with related pages seen (range 0.146 - 0.258 with 95% confidence). No significant changes in this trend were observed over time.

On desktop, the clickthrough rate for any related link given is 0.0117 clicks per page (+/- 0.0066, 95% confidence). When users view the related pages (by scrolling to the bottom of the article) the clickthrough rate is 0.0278 per page with related pages seen (+/- 0.0168, 95% confidence). No significant changes in this trend were observed over time.

• comparison between mobile and desktop
• 
  On mobile, almost 8% of all pageviews lead to a user clicking on a related page link. For all pages in which the users scrolls to the bottom, the likelihood is 20%. On desktop however, the interest in related page is much lower and did not meet our stated goal for the feature.
• 
  Over time, the click through rate on mobile has stayed constant at ~20%, indicating that users appreciate the links. If the links were not helpful, we would see a decline in click-through-rate over time.

Query:

SELECT count(*) AS num_Events, event_eventName,

LEFT(TIMESTAMP, 8) AS dt, event_skin

FROM RelatedArticles_14496900

GROUP BY dt, event_eventName, event_skin;