MediaWiki

Reading/Multimedia/Media Viewer/Research 2014

< Reading | Multimedia | Media Viewer

This report summarizes research findings for Media Viewer, a new multimedia browser aimed at Wikimedia readers and casual editors. This tool was developed by Wikimedia Foundation's multimedia team from July 2013 to December 2014 (see project timeline).

Check out the Media Viewer Research 2014 Slides, for a visual presentation of key findings


Key findings include image views, enabled rates, activity and performance metrics, as well as to usability tests and user feedback. This research was conducted to inform product development and learn more about the tool's usage, speed and overall impact.

See companion slides for a visual presentation of the research outlined below.

Overview

edit

Goals

edit

Media Viewer aims to:

  • improve the viewing experience for readers
  • make it easier to preview and browse images
  • show basic information, link to more details
  • help you share, download or embed images

Features

edit

 
This annotated screenshot shows key features of Media Viewer's final user interface.
Ahu-Akivi Photo by Ian Sewell licensed under CC-BY-SA-3.0 via Wikimedia Commons.

Here are some of the features developed to support goals and user needs for Media Viewer:

The effectiveness of these features was tracked through a range of research tools, from behavioral metrics to usability tests, as described below.

Research

edit

We did extensive research throughout Media Viewer's development and after its release in 2014, to learn more about the tool's usage, performance and impact.

Research methods included:

  • activity metrics (e.g.: image views)
  • enable/disable rates (e.g.: % opt-outs)
  • performance metrics (e.g.: load times)
  • design research (e.g.: usability testing)
  • user feedback (consultations, survey, talk pages)

Throughout the year, research metrics and user feedback guided our product development, at each step of the way. Read on for more detailed findings on each method.

Key Findings

edit

Here are our key findings from Media Viewer research in 2014:

Image Views

edit

Image Views - Top Sites

edit
 
Global daily image views in Media Viewer, as well as for the top 6 Wikimedia sites, Nov.-Dec. 2014. Source: Wikimedia Foundation

Global image views average about 23 M views per day: they fluctuate between 20M and 26M/day, with fewer views on weekends.

The top 6 sites for image views as of Dec. 7 were:

  • English Wikipedia (9.0M)
  • German Wikipedia (2.7M)
  • Russian Wikipedia (2.0M)
  • French Wikipedia (1.6M)
  • Spanish Wikipedia (1.0M)
  • Wikimedia Commons (1.2M)

Monthly image views (712M) represent about 3.4% of monthly page views (20B) on all our sites.

Note that many these image views may not have been intentional, such as people pressing the previous button to go back to an earlier image (see next section).

Media Viewer vs. File Page

edit

 
Daily image views for Media Viewer and File Page, May-Dec. 2014. Source: WMF

Global image views for Media Viewer were about 23 million per day in November 2014, versus 3.4 million for file pages, for a combined total of 27 million daily image views.

Total image views nearly doubled from May to December, 2014, with 87% coming from Media Viewer and 13% from the file page. (Note that many these image views may not have been intentional, such as people pressing the previous button to go back to an earlier image, as described in the next section.)

During that period, file page views declined by 73%, from about 12 million in May, before Media Viewer was launched on English and German Wikipedias, down to about 3 million by the end of 2014.

Note that these metrics do not include the variance from earlier releases on other large wikis like the French or Spanish Wikipedias and Wikimedia Commons, as we did not collect all these metrics prior to May 2014.

So the actual variance is likely to be higher, but would require more research to determine.

Intentional Views

edit

Not all Media Viewer image views can be considered 'intentional' -- here's one way to separate them from each other:

'Intentional' views (~17M/day)

  • thumbnail + magnifier clicks
  • next clicks
  • hash loads
  • file page views

'Unintentional' views (~6M/day)

  • previous clicks
  • history navigation events

On that basis, 'intentional' image views can be estimated at about 17M (versus ~6M for 'unintentional' views).

Overall, how often users look at images increased significantly with the deployment of Media Viewer - the sum of file page visits plus 'intentional' Media Viewer views, grew by about 50% after the release on English and German Wikipedias.

(This does not include the growth from the earlier releases on other large wikis like the French or Spanish Wikipedias and Wikimedia Commons as we did not track file page visits at the time. So the actual growth is likely to be higher, but would require more research to determine.)

Enabled Rates

edit

 
Percent of global logged-in users with Media Viewer enabled or disabled as of Dec. 8, 2014, by user group. Source: WMF.

Media Viewer was enabled for about 99.54% of logged-in visitors to Wikimedia sites as of December 8, 2014 (versus 0.46% disabled).

Those rates are lower for advanced users:

  • 95.57% of active contributors had Media Viewer enabled (users with 5+ edits / month)
  • 85.91% of very active contributors had Media Viewer enabled (users w/ 100+ edits / month)

These numbers were collected a month after a very prominent enable/disable button was added to Media Viewer, to make it easier for users who didn't want the tool to opt-out. The base group for each percentage is user accounts with a user_touched date or 5 edits or 100 edits in the last 30 days. (Local accounts belonging to the same global account are not deduplicated.)

 
Cumulative opt-out rates for logged-in users show more active users disabled Media Viewer from Aug. to Dec. 2014. Source: WMF

Enabled/disabled rates seem to be stabilizing across all logged-in user groups.

Opt-ins & Opt-outs - Logged-in vs. Anonymous

edit

 
Global opt-out events by anonymous users started in July 2014 and varied as new features were released, with a downward trend by year-end. Source: WMF

Anonymous opt-outs and opt-ins increased in early September, after File Usage was removed, making the Disable/Enable links easier to find.

After the Enlarge feature was released in late September, global anonymous opt-outs dropped by about 60%, from 2.2k opt-outs per day down to 1k. The Enlarge feature addresses one of the most frequent user requests, enabling users to zoom in on images.

After a more prominent Disable/Enable button was released in November, anonymous opt-outs events nearly tripled, from 1k back up to 3k opt-outs per day -- and opt-outs by logged-in users also increased to an average of 100 opt-outs/day.

By the end of 2014, opt-outs for both anonymous and logged-in users had dropped by half again, from a peak of 3.6k down to 1.9k total opt-outs per day.

Opt-outs by logged-in users surged briefly in late August, after the default for Wikimedia Commons was switched to opt-in for logged-in users, to facilitate curation on that multimedia file repository.

Other feature changes did not seem to impact opt-outs by logged-in users, which were steady at about 65 opt-outs/day by year-end.

Opt-ins by anonymous users seemed less impacted by feature changes, representing about one-sixth of opt-outs by the end of the year. (Note that opt-ins by logged-in users is not shown, due to a bug).

Note that for technical reasons, it is quite hard for anonymous users to re-enable Media Viewer on most wikis, unless they do so right after disabling. So the opt-in events on this chart may be due to users trying out the disable feature and choosing to re-enable it.

Activity Metrics

edit

Here are some of the user actions tracked through EventLogging in 2014. Please refer to the image in the Features section above to see how these actions are shown in Media Viewer's final user interface.

Top 5 Actions

edit
 
Global daily clicks for top 5 actions in Media Viewer, Nov.-Dec. 2014. Source: Wikimedia Foundation

Here are the top 5 actions for Media Viewer, tracked by number of daily clicks:

  • Open/Thumbnail clicks (10.3M)
  • Close (9.1M)
  • Next (6.8M)
  • Previous (2.5M)
  • History (4M)

Every day, we log an average of 10 million clicks on thumbnails in articles -- and almost as many clicks on the Next and Previous buttons. This new browsing feature did not exist before Media Viewer. Its widespread use suggests that Media Viewer may be adding value for millions of readers, who would not be using this browsing feature if they did not find it useful.

History tracks how often users navigate within Media Viewer using their browser's history.

Enlarge & Full Screen

edit
 
Global daily clicks for Enlarge, Full Screen and related actions in Media Viewer, Nov.-Dec. 2014. Source: Wikimedia Foundation

To address frequent requests for a 'zoom' function, a new 'Enlarge' feature lets you click on an image to view its original file in full resolution: it is now used 12x more often (1M/days) than the full screen feature (93k/day), as of December 7, 2014. We also observed a dramatic decline in opt-out rates after this new feature was introduced (as shown in a separate graph).

This data and our design research suggest that the Enlarge feature significantly improved overall user experience by enabling users to view images in detail. In usability studies, the full screen feature is still found useful, because it lets you fill the screen with the selected image -- but not as useful as zooming onto the image.

Metadata & Details

edit
 
Global daily events to open/close the metadata panel, or view 'more details' on file pages in Media Viewer, Dec. 2014. Source: Wikimedia Foundation

This chart compares global daily events for opening (or closing) the metadata panel, versus clicking on 'more details' to go to file description pages: the metadata panel is now used 400x more often (1.3M/day) than the 'More details' button (~67k) in Media Viewer. Note that this total includes scrolling events to open the metadata panel (1.2M/day), as well as keyboard presses on the up arrows (130k).

It's also worth noting that the metadata panel open frequency is around 7% of image view frequency, so people actually use the panel fairly often (way more than the 'More details' button to access the file description page, which is ~0.25%).

Overall, this data suggests that the metadata panel is valuable to users.

Download, Disable & Share

edit
 
Global daily clicks for Download, Disable & Share panel actions in Media Viewer, Nov.-Dec. 2014. Source: Wikimedia Foundation

The Download panel is opened more often (74k / day) than the Disable/Enable panel (26k / day -- labeled 'options' in this graph) or the Share/Embed panel (20k / day -- labeled 'use this file' in this graph).

edit
 
Global daily clicks for 9 information-related links in Media Viewer, Nov.-Dec. 2014. Source: WMF

In the information that appears below the image, users click more often on Source links (7k/day) than on Author links (3k/day).

They also click a bit more on 'View terms' (4k/day) than on License links (3k/day - e.g. 'CC-BY-SA').

Predictably, links that are below the fold (e.g. Uploader, About) get fewer clicks than above the fold, because users have to scroll down or click on an ellipsis to open the metadata panel that contains them. It's also worth noting that clicks on source and terms decreased by half after we launched the captions above the fold, perhaps because that new information in the first two lines detracted from those links on the third line.

New Features

edit

 
Global clicks on 'Enlarge', 'More details' and 'Download' surged after launch in Sep.-Oct. 2014. Source: WMF

After the new Enlarge feature was released, we observed a dramatic 20x increase in clicks for this popular 'zoom' feature, one of the most frequent user requests. By clicking anywhere on the image, users can now go to the original file and use their browser's zoom tool to enlarge image details.

The release of the the big blue 'More details' button also registered a 3x increase in clicks, compared to the Commons icon that was featured before (both link to the file description page).

And giving the Download feature its own icon increased clicks by 3x as well (it used to be a tab in a single 'Use this file' panel, along with 'Share' and 'Embed').

Performance Metrics

edit

Media Viewer vs. File Page

edit

 
Global image load times for Media Viewer vs. File Pages are on par, for either median or 95th percentile, Dec. 2014. Source: WMF

This chart compares image load times between Media Viewer and the File page in Dec. 2014.

It suggests that load times are about the same, with a median load time of 1.5 seconds to 2 seconds -- even though Media Viewer typically loads larger images than the File Page.

For the 95th percentile (the longest average load for 95% of users), both viewing methods are also on par, with load times in the 8-10 seconds range. Note that File Page performance improved significantly after WMF implemented a switch to HHVM in November 2014: prior to that, Media Viewer was loading images about twice as fast as the file page.

Network Performance

edit
 
Network performance for Media Viewer image load times, in milliseconds - Dec. 2014. Source: WMF

This chart shows the average duration of image loads per country, in milliseconds, as a reflection of network performance. Faster load times are shown in green, slower load times in red.

Overall network performance seems related to Internet bandwith in each region, with faster load times in developed countries than in the developing world.

More metrics

edit

Attributions & Licenses

edit
 
Percent of images with author, source or license information in Media Viewer, Nov.-Dec. 2014. Source: WMF

A growing number of images shown in Media Viewer now have machine-readable data for author, source or license information.

These ratios have been increasing steadily, as a percent of image views: 97% now have license information, 86% have sources and 86% have authors.

These improvements are related to a recent drive to clean up metadata for multimedia files across Wikimedia sites.

Design Research

edit

The Wikimedia Foundation's design research team conducted a range of design research studies and usability tests for this project:

User Study 1

edit

Here are key issues discovered in our first design research study in July 2014:

  • Hard to discover info panel
  • Info panel covers images & is hard to control
  • Hard to find 'View original file' button
  • Hard to find 'Download' button
  • Clarify meaning of viewing options

See User study 1 slides (July 2014).

User Studies 2 and 3

edit

In studies conducted in August-September 2014, users successfully completed all tasks listed below:

Primary Tasks

  • Preview images
  • Learn about images
  • Get more details
  • Browse images
  • Disable feature (passed 2nd test)
  • Re-enable feature (passed 2nd test)

Secondary Tasks

  • Enlarge
  • Share
  • Download
  • Go back

See User Study 2 Slides (August-September 2014).

User Study 4

edit

In a final user study conducted in November 2014, we observed that readers were able to accomplish these key tasks with no problems:

  • View images in Media Viewer
  • Navigate to next / previous image
  • Read a long description
  • Identify details about images
  • Get more details on file pages
  • Zoom in to see details
  • View image in full screen
  • Share an image with a friend
  • Disable Media Viewer
  • Re-enable Media Viewer

See User Study 4 Report (November 2014).

User Feedback

edit

Feedback channels

edit

User feedback came from a variety of channels:

Talk Pages

edit

Here are some key issues reported in talk pages after release:

  • Make it opt-in
  • Hard to find file page link
  • Show caption higher up
  • Can't go back quickly
  • Image load too slow
  • Issues on tablets/mobile
  • Show full file name
  • Show more license info
  • Show label for author
  • Make image clickable

See Media Viewer Talk Page.

Requests for comments (RfCs)

edit

Three community RfCs were held from June to October 2014:

All RfCs asked that Media Viewer be disabled by default.

For the most part, the WMF did not fulfill these requests, due to concerns about the lack of representation, but committed to more improvements, based on community consultation (see next section).

Note that the Foundation partially implemented the Commons RfC request, acknowledging the differences in usage patterns between Wikimedia Commons and "reader-facing" wikis: Media Viewer's default configuration was switched to 'opt-in' for registered users on that site in August 2014.

Community Consultation

edit

A community consultation was held on Meta, from August 28 to September 7:

  • the goal was identify any critical issues
  • it was widely promoted on community channels
  • community members made 130 suggestions
  • WMF evaluated and prioritized all suggestions
  • WMF selected 10 'must-have' tasks
  • WMF developed 13 new improvements
  • all improvements were released and validated by Nov. 2014
  • see improvements list below

See Consultation Page.

Improvements

edit

These improvements were made from Sep. to Nov. 2014, based on community consultation:

See Improvements Page.

Surveys

edit

Users were asked for feedback in a short survey (April-July 2014):

  • optional survey (subject to self-selection bias)
  • built-in feedback button (opened survey popup window)
  • 8 different languages (ca, de, en, es, fr, hu, nl, pr, pt)
  • 18K+ survey responses (satisfaction ratings, some with comments)
  • a majority found the tool useful (but sample may not be representative)
  • a lot of actionable feedback (hand-coded to identify and prioritize key issues)

See Survey Report.

Survey feedback

edit

Here are top feature requests from surveys:

  • Want zoom 14%
  • More image sizes 8%
  • Want opt-out 6%
  • Easy way to exit 5%
  • Scroll down for info 5%
  • Info cuts off images 5%
  • Can't find info 4%
  • Show whole image 3%
  • More info above 3%
  • Disoriented by UI 3%

(% hand-coded survey requests)

See Survey feedback.

Survey results

edit
 
A majority of global respondents found the tool useful -- but this was an optional survey and the sample may not be representative. Also, this global chart adds together feedback from different sites with different response rates and survey running times. Source: WMF

Here are the final survey results, based on 18,199 global responses as of July 8, 2014, when all surveys ended.

Note that this was an optional survey, so approval rates should not be cited as a conclusive metric, as they are subject to self-selection bias. Editors are vastly overrepresented (half of the respondents self-identified as editors), the volume of responses is much higher in the first few days following deployments, and smaller wikis are overrepresented since the surveys have been running for longer there. The primary purpose of the survey was to collect user comments in order to improve the tool -- not to use approval ratings as a definitive measure of success.

Keeping these caveats in mind, 56% of global respondents found the tool "useful for viewing images and learning about them", with 35% responding "not useful" and 9% "not sure", as shown on this spreadsheet.

 
Reader survey responses in 7 languages, from 14 April to 7 July, 2014. Source: WMF

Readers responded more positively than editors, with 65% of them finding Media Viewer useful, based on 7,582 reader survey responses in 7 languages, over a 12-week period, from 14 April to 7 July, 2014 (see linked spreadsheet).

Weekly reader approval rates are shown here for each language. The drop in non-English rates in May-June may be due to non-English readers of Enwiki and Commons: the surveys are based on interface language, not content language.

Cumulative "useful" approval ratings from all users varied greatly by language: English 36%, French 70%, Spanish 78%, Dutch 59%, Portuguese 81%, German 30%, Hungarian 63%, Catalan 71%.

Overall approval rates from English users gradually increased in the month following launch: the weekly percentage of survey respondents who found Media Viewer useful ("yes") grew from 26% in May to 52% in June 2014. Note that the survey response rate fell at the same time, as shown in these results.

Again, note that this was an optional survey, so approval rates should not be cited as a conclusive metric, as they are subject to self-selection bias. They are only shown here for the record, as they were frequently referred to in community discussions.

edit

Thanks

edit

This research was collected and analyzed for over a year by many different individuals, who worked diligently, above and beyond the call of duty, to help the development team make more informed decisions. We are very grateful to everyone who made this possible!

Research contributors include: Aaron Arcos, Daisy Chen, Gilles Dubuc, Fabrice Florin, Pau Giner, Mark Holmquist, Keegan Peterzell, Abbey Ripstra, Gergo Tisza.

Research advisors include: Howie Fung, Aaron Halfaker, Rob Lanphier, Erik Moeller, Toby Negrin, Dario Taraborelli, Lila Tretikov.

This report was prepared by Fabrice Florin and Gergo Tisza, on behalf of the team, from Dec. 2014 to Jan. 2015.

Retrieved from "https://www.mediawiki.org/w/index.php?title=Reading/Multimedia/Media_Viewer/Research_2014&oldid=4436633"