Page Previews

Group:	Reading
Start:	2014-03-26
End:	2018-05
Team members:	Prateek Saxena, Quiddity (WMF), Chris Koerner, Jon Katz, Jon Robson, Joaquin Oltra Hernandez, Jeff Hobson, Tilman Bayer, Piotr Miazga, Nirzar Pangarkar, Sam Smith, Sherah Smith, Vibha Bamba, Jared Zimmerman
Lead:	Olga Vasileva

Translate this page

For the extension that powers this feature, see Extension:Popups .

The Page Previews feature solves the core problem of users needing to open multiple tabs to gain an understanding of a word or concept within the context of the subject they are reading. With Page Previews, whenever a reader hovers over a link to another article, a short summary of the subject and an image (if available) is displayed. The user can then decide whether they wish to visit that subject more thoroughly before continuing with the current article. A complete description of Page Previews functionality is available. An overview over all related technical documentation is provided in this blog post from May 2018.

The Reference Previews feature is a related project. It shows a preview of a footnote whenever a reader hovers over its footnote marker. A description of this feature can be found on its project page.

Introduction

Page Previews are designed to reduce the cost of exploration of a link, as well as to promote learning by allowing readers to gain context on the article they are reading or to define an unfamiliar term, object, event, or idea without navigating away from their original topic. For casual readers, Page Previews will make it easier for users to get an overview of an article before deciding whether or not to browse to it. Users interested in reading an entire article in its totality will not be distracted or discouraged by viewing unfamiliar concepts—they can simply preview the concept without navigating to a new page. Consequently, the smoothness of the user experience for these users will be increased.

There have been many requests over the years for a similar feature; there are browser extensions (see list) and one heavily used gadget (the more editor-focused Navigation popups) that also exist to solve this problem. Page Previews were first built as a beta feature in 2014 (called Hovercards) and have been among our most popular beta features in terms of adoption, user feedback and impact on usage. On the primary talk page for Hovercards, over the course of 2 years, numerous issues across all projects were identified and resolved. The mobile equivalent of Page Previews, implemented on the Android app in September 2015, led to a 20% increase in links clicked per page.

The Page Previews settings panel enables logged-out users to enable and disable the feature based on their preferences. Logged-in users may enable or disable Page Previews from the Appearance section of Special:Preferences. If Navigation popups and Page Previews are enabled at the same time (typically, for logged-in users who have opted in via the beta feature), Navigation popups takes precedence. To enable Page Previews, Navigation pop-ups must be disabled.

Example of a Page Preview

Design

For the current iteration of Page Previews, each preview contains the following:

A portion of the first paragraph from the article
An image (if available) from the article. Images appear horizontal or vertical depending on the location of the link within the article.
A settings cog which allows users to turn Page Previews on and off

Example of Page Previews in Greek
Example of Page Previews in Chinese
Example of Page Previews in English

Key differences from Navigation Popups

Navigation popups is for power users and has many features that are not relevant to most site users, such as quick links to meta-data pages like History, Diffs, and Usercontributions.
Navigation popups work for links to all namespaces, and on all Wikimedia projects. Page Previews is configured to only work with mainspace links, and currently only on Wikipedias.
Page Previews are styled to enable easy reading. Currently, Nav-popups' type-size is small and the margins are tight.
Page Previews are available to all users, not just logged-in users.
Page Previews use a visual style consistent with the article type treatment
Page Previews emphasize the lead images, so a user gets an idea of a term using both text and image.
The action set in the gadget feels out of context, hence we need to validate which of those actions are useful for readers and editors.

HTML Previews

As of March 1st, 2018, page previews now display previews in HTML. This resolves many outstanding bugs, provides correct representation of mathematical, chemical and other formulas, and ensures previews display the same formatting as the article content.

Settings

Enable/Disable

Logged-Out Users

Logged-out users may turn Page Previews off via the cog-icon displayed at the bottom of each preview. If a user wishes to re-enable Page Previews, they can be enabled via the "Edit preview settings" link available at the bottom of any wiki page. They can also disable JavaScript in their browser to turn it off permanently.

Select setting cog to disable Page Previews
Select disable to disable the feature
Instructions on how to re-enable will be presented

The link to re-enable Page Previews is available at the bottom of each page

Logged-In Users

Logged-in users can control their settings from two places. They may enable or disable Page Previews from the Appearance section of Special:Preferences. Logged-in users may also turn Page Previews off via the cog-icon displayed at the bottom of each preview.

If a logged-in user clicks on the settings cog, the system will redirect them to the user preferences page.
If a user previously had the Hovercards beta feature enabled, their preference will be kept and appear in the Appearance section of user preferences.

Users may enable the feature by using the settings cog at the bottom of each preview, or by selecting the option directly from the user preferences page

Success Metrics and Feature Evaluation

A number of qualitative and quantitative tests were performed to evaluate the performance of the Page Previews feature. These tests were focused on the following questions:

Do users enjoy Page Previews and find them useful?
How do Page Previews change reading behavior and do they help users be more accurate when selecting the articles they wish to read?
Would launching the Page Previews feature have effects on fundraising?

2015 Greek and Catalan Wikipedia test

A feature test was implemented for a period of 4 months on Greek and Catalan Wikipedias. A number of issues and bugs were reported and user satisfaction was recorded using a survey. Users had generally favorable feedback, with the majority of users finding Page previews (Hovercards) useful, easy to use, and enjoyable to use.

Read the full results of these tests.

2016 Qualitative test

To determine user attitudes towards the Page Previews feature and trace further changes in reader behavior, an unmoderated remote panel study was performed using UserZoom software. The majority of participants reported positive attitudes to the Page Previews feature. In addition, the majority of participants had no issues in turning the feature on and off. Users also described the feature in generally positive terms and reported it was not distracting to their reading experience.

Read the full results and analysis of this test.

2016 A/B tests on Hungarian, Italian, and Russian Wikipedias

To determine changes in reading behavior and evaluate the success of the Page Previews (Hovercards) beta feature, three A/B tests were launched on Hungarian, Italian, and Russian Wikipedias. The A/B test on Hungarian was launched June 7, 2016, the A/B tests on Italian and Russian were launched Sept 23, 2016.

The results of these tests indicated that Page Previews facilitate positive changes in reading behavior by increasing the precision with which users select the pages they read, reducing the cost of exploration of other pages, and allowing users to selectively focus on a single topic by providing context within a page. During the process of collecting and analysing our data we ran into a number of bugs and issues within our instrumentation that allowed us to refine and improve, to ensure subsequent tests were issue-free.

Read the full results and analysis of these tests.

2017-2018 A/B tests on English and German Wikipedias

From October to November 2017 and December 2017 to February 2018, we ran two A/B tests to gauge the performance of the feature prior to release on these projects. The results, described in more detail in the report, displayed a decrease in pageviews, but an overall increase in the number of distinct pages that readers interact with, indicating that they are more likely to explore a larger variety of topics when the feature is on. We also noted decreased usage of the back button and very low rates of disabling the feature. In addition, we ran a number of fundraising tests on enwiki.

Improvements since English Wikipedia discussion from April 2016

During the discussion from April 2016, a number of bugs, improvements, and other requests were identified which we have addressed in the following ways:

Previews “getting stuck” in the open position

We have since refactored our code and this is no longer an issue

Bolded text and mathematical formulas not appearing within the preview

Currently, previews display in HTML and content is true to the article content

A means to disable the feature from the card itself

Logged-in and logged-out users can now disable the feature by selecting the settings gear within the card itself.

A means to make sure page previews do not interfere with navigational popups

Logged-in users may either have navigational popups or page previews. If you’re a navpopups user, you will not be able to enable page previews unless you turn off navpopups first, and vice versa

Providing more thorough research and A/B test results, performing a large-scale A/B test on English Wikipedia

Since the discussion, we have performed two series of A/B tests, the first on Hungarian, Russian, and Italian Wikipedias. And the second on English and German Wikipedias.

Limiting performance impact

We have worked with our performance team to minimize any issues page previews. A dashboard with detailed information on feature performance was created.

Providing a way to enable the feature for anonymous users only

This is ready as well. As of March 2018, the feature is enabled by default for anonymous users on all Wikipedias except English and German

Providing data on how often users disable the feature

For both of our 2017/18 A/B tests, the disable rate was around 0.01%. Because the rate had been similarly low in earlier A/B tests, we also did a qualitative test to confirm that users did not have problems enabling or disabling the feature. Our conclusion is that people know how to turn the feature off without difficulty but do so at extremely low rates.

Providing information on whether the feature was considered a nuisance

Based on our qualitative testing, we concluded that tested users did not find the feature distracting, or a nuisance.

Disabled pages

For security reasons, Page Previews won't be loaded on certain special pages. We refer to these pages as "disabled pages". The initial version of this can be found here: task T170893.

Future Iterations and Potential Improvements

Following the general launch of Page Previews, we are planning on iterating on the feature to add functionality. These items will include:

Configuration of images and other settings:
- a number of users have requested making images configurable within Page Previews. Request declined: T148995
- other logged out preferences (more info on preferences)
Article titles - currently, only the summary of the article is available within Page Previews. As an iteration, we would like to display the title of each article along with its summary
Reference Tooltips (phab:T67114). The project page for the Reference Previews feature has more details.

Using Page Previews to gain more information on each reference without scrolling to the bottom of the page

Cross-wiki Page Previews (phab:T67117), and subsection links (phab:T65792), particularly for Wiktionary
Integrate with Navigation Pop-ups - we would also like to expand the configuration of Page Previews to include some of the functionality planned in navigation pop-ups.
Interacting with Page Previews text - a number of requests for interacting with preview text have been filed, including being able to copy and paste the text or export the text as a form or article summary. More info here (phab:T146097)
Integrating with Wiktionary items to provide definitions on hover.

Interacting with Wiktionary items to provide definitions for unknown

FAQ

Why Page Previews?: This is a feature intended to improve the experience for any reader who normally would have clicked on a blue link in Wikipedia because they needed an overview (definition) of that entity. It's inspired by one of the most popular gadgets, Navigation popups.

How do we measure Page Previews performance?

The theory of impact for Page Previews is that they lower the cost of exploring a link. This should mean that users are less inhibited and more focused when exploring links. We should see that the overall links clicked + (non-accidental) hovers exceed the number of links clicked without Page Previews. This is what success looks like. There are also some indications of failure that we will look for:

hovers can be accidental - we need to measure normal dwell time in a controlled condition to ensure that the likely rate of accidental hovers is not too high. To give an example, if a user must dwell on a link for 250 msecs before a hover shows, then we would want to make sure that there are not a large number of users who tend to dwell on a link for more than 250 msecs without clicking it.
Page Previews could lead to fewer page views, because the user gets the information that they need from the preview- this is not a problem, but we want to make sure that the decrease (if any) does not result in significantly less editing or fundraising.
the % of hovers that result in a user continuing to the page is high - this would suggest that most people wanted to go to the page anyway. If this is the case, then the hover is likely just adding an unnecessary step. We expect some significant % of click throughs for hovers. It is ~60% on Android for a similar feature, but we expect it to be lower on desktop.
the percentage of users who disable Page Previews is low (given that users are aware on how to disable the feature) - this suggests that users enjoy the feature.

Is this enabled by default for logged-in users?: No. If you had the feature enabled as a Beta feature you will still have the feature enabled. See Preferences>Appearance>Reading preferences to change the setting.
What if I have Navpops enabled by default?: If you have Navpops enabled, Page Previews are automatically disabled. You will have to disable Navpops in order to experience Page Previews. This is an intentional decision to ensure that Navpop users' do not have their preferences interrupted. Note: for certain browsers, you might have to clear your browser cache first for the change to take place.
How many options do I have in Page Previews preferences?: Right now, Page Previews will either be turned on or off by a user. The option to turn them off lives in each hover event. So at each hover event, a user can decide that they are no longer interested. For logged-in users, Page Previews may be re-enabled from user settings under the section titled "Appearance". For logged-out users, Page Previews may be re-enabled by selecting the "Enable Previews" link at the bottom of the page.

At the project level, administrators can determine how long a user should dwell on a link before a hover is triggered. If a project wants to be conservative, the lag can be longer. WMF will have a recommendation as to optimum dwell time that provides the best user experience while minimizing accidental hovers.

Why can't users just turn on Page Previews if they want them?: Unfortunately, there is no good way to tell users about a new feature without showing it to them first.

Central notice banners have been suggested, but running them for 2 weeks would not solve the needs of future users and we do not want to run them continuously. Notifying a user of a new feature is best done using a...hover-over.

What impact to Page Previews have on page views?: According to A/B tests we did not see a large decrease in session depth across all Wikipedias. Average session depth (average number of pages viewed per session) remained relatively equal between the Previews on and off groups.
What about accessibility?: Page Previews only activate when the focus/hover state occurs over a link. They are usable for people who use keyboard navigation. For screen readers, Page Previews have proper WAI-ARIA semantics declaring them as tooltips. In short, screen reader software ignore Page Previews.
If I have Page Previews feedback or if I have a suggestion for making them better, where should I go?: Please go to the Page Previews discussion page.
The Page Previews continues to show an old version of a page. What can I do?: The Page Preview shows a cached version of a page. If the page has been edited and you want Page Previews to show the new version right away, you can purge the page.
How can I exclude certain page images?: See Extension:PageImages for details.

Code

Analytics and instrumentation

Page Previews were instrumented to measure various aspects of usage. Please see the Popups extension page for details and feel free to ask any clarifying questions.
To measure content usage via Page Previews, in an aggregated way that is compatible with the existing Pageviews data, a separate instrumentation was set up that went live in April 2018. There are plans to make some of this data public for the benefit of e.g. Wikipedia editors and academic researchers.

Project-level rollout discussions

Rollout Update July 2018

Page previews are now available on all Wikipedias. The feature is on by default for anonymous users and newly created accounts and off by default for accounts created before July 10, 2018.

Rollout Update April 2018

The next step of the rollout for English and German Wikipedias is planned for the first half of April 2018. This deployment will include turning the feature on by default for logged-out users. This will mean no changes for logged-in users. The feature will be off by default for logged-in editors, unless currently enabled. If you would like to enable it, it is available in your Preferences under “Appearance”. If you have the feature enabled already, it will stay on.

In terms of future changes for logged-in users, we have a few options we will be requesting feedback on:

Keep the feature off by default for logged-in users.
Turn the feature on by default for new accounts only. Currently, when users move from being readers to contributors and create an account, the feature will seem to vanish, and that would be confusing. As a further step in the feature rollout, we plan to change this configuration and enable the feature for all new accounts.
Turn the feature on by default for existing logged-in users (Even if it were enabled for everyone, it would still be automatically suppressed for anyone who uses NAVPOPS.)

Rollout Plan

We began rolling out to Catalan, Greek, Hungarian, Italian, and Russian Wikipedias in early 2017. Data from the A/B tests represented highly favorable views of the feature. After this, we would like to continue rolling out in stages to other Wikipedias as described in the table below.

Code	Language	Phase	Onboarding	Success	Dates
ca	Catalan	0	^[1]	✔	March 2017
el	Greek	0	^[2]	✔	March 2017
it	Italian	0	^[3]	✔	March 2017
he	Hebrew	0	^[4]	✔	March 2017
hu	Hungarian	1	^[5]^[6]	✔	March 2017
-	All other Wikipedias except the remaining top 6	1	^[7]	✔	July 2017
ja	Japanese	1	^[8]	✔	August 2017
es	Spanish	1	^[9]	✔	August 2017
fr	French	1	^[10] ^[11]	✔	August 2017
de	German	2	^[12]	✔	April 2018
en	English	2	^[13]		April 2018

Onboarding Experience

The onboarding experience for the feature will depend on the consensus of each community on having Page Previews on by default or providing users with an onboarding experience during which they can reject the feature.

References

[1] ttps://ca.wikipedia.org/wiki/Tema:Thijlqz8uwve1490

[2] ttps://el.wikipedia.org/wiki/Βικιπαίδεια:Αγορά/Αρχείο_2016/Δεκέμβριος#Αναδυόμενες_κάρτες

[3] ttps://it.wikipedia.org/wiki/Wikipedia:Malfunzionamenti#Hovercards

[4] <https://he.wikipedia.org/wiki/ויקיפדיה:מזנון/ארכיון_387#הצעה:_הפעלת_כרטיסים_מרחפים_בוויקיפדיה_בעברית

[5] ttps://hu.wikipedia.org/wiki/Wikipédia:Kocsmafal_(műszaki)/Archív104#Információs_kártyák

[6] ttps://hu.wikipedia.org/wiki/Wikipédia:Kocsmafal_(műszaki)/Archív105#Információs_kártyák_(másodszor)

[7] this message was sent to Wikipedia Village Pump (or equivalent) of the 247 wikis included in this phase on 19 April

[8] ttps://ja.wikipedia.org/wiki/Wikipedia:井戸端#New_Page_previews_feature

[9] ttps://es.wikipedia.org/wiki/Wikipedia:Café/Archivo/Noticias/Actual#Nueva_funci.C3.B3n_Vistas_previas_de_p.C3.A1ginas

[10] ttps://fr.wikipedia.org/wiki/Wikipédia:Le_Bistro/19_avril_2017#Nouvelle_fonctionnalité_Aperçus_des_pages

[11] r:Wikipédia:Le Bistro/21 juillet 2017#Page Previews (Hovercards) update

[12] :Wikipedia:Fragen_zur_Wikipedia/Archiv/2018/Woche_14#Seitenvorschaubilder (Page_Previews)

[13] ttps://en.wikipedia.org/wiki/Wikipedia:Village_pump_(miscellaneous)#Page_Previews

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]