Reading/Multimedia/Media Viewer/Research 2014



This report summarizes research findings for Media Viewer, a new multimedia browser aimed at Wikimedia readers and casual editors. This tool was developed by Wikimedia Foundation's multimedia team from July 2013 to December 2014 (see project timeline).

Key findings include image views, enabled rates, activity and performance metrics, as well as to usability tests and user feedback. This research was conducted to inform product development and learn more about the tool's usage, speed and overall impact.

See companion slides for a visual presentation of the research outlined below.

Goals
Media Viewer aims to:
 * improve the viewing experience for readers
 * make it easier to preview and browse images
 * show basic information, link to more details
 * help you share, download or embed images

Features
This annotated screenshot shows key features of Media Viewer's final user interface. Ahu-Akivi Photo by Ian Sewell licensed under CC-BY-SA-3.0 via Wikimedia Commons. 

Here are some of the features developed to support goals and user needs for Media Viewer:
 * enlarge the image
 * view the image in full screen
 * browse through related images
 * read the image caption or description
 * learn more about the author, source or license
 * get more details or edit information on the file page
 * share, embed or download an image
 * close Media Viewer
 * disable (or enable) Media Viewer

The effectiveness of these features was tracked through a range of research tools, from behavioral metrics to usability tests, as described below.

Research
We did extensive research throughout Media Viewer's development and after its release in 2014, to learn more about the tool's usage, performance and impact.

Research methods included:
 * activity metrics (e.g.: image views)
 * enable/disable rates (e.g.: % opt-outs)
 * performance metrics (e.g.: load times)
 * design research (e.g.: usability testing)
 * user feedback (consultations, survey, talk pages)

Throughout the year, research metrics and user feedback guided our product development, at each step of the way. Read on for more detailed findings on each method.

Key Findings
Here are our key findings from Media Viewer research in 2014:
 * Media Viewer serves a lot more images than before (23M views/day)
 * Most users keep Media Viewer enabled (99.5% enabled)
 * Media Viewer key features were found easy to use
 * Media Viewer is more useful for readers than editors

Image Views - Top Sites


Global image views average about 23 M views per day: they fluctuate between 20M and 26M/day, with fewer views on weekends.

The top 6 sites for image views as of Dec. 7 were:
 * English Wikipedia (9.0M)
 * German Wikipedia (2.7M)
 * Russian Wikipedia (2.0M)
 * French Wikipedia (1.6M)
 * Spanish Wikipedia (1.0M)
 * Wikimedia Commons (1.2M)

Monthly image views (712M) represent about 3.4% of monthly page views (20B) on all our sites.

Note that many these image views may not have been intentional, such as people pressing the previous button to go back to an earlier image (see next section).

Media Viewer vs. File Page
''Daily image views for Media Viewer and File Page, May-Dec. 2014. Source: WMF''

Global image views for Media Viewer were about 23 million per day in November 2014, versus 3.4 million for file pages, for a combined total of 27 million daily image views.

Total image views nearly doubled from May to December, 2014, with 87% coming from Media Viewer and 13% from the file page. (Note that many these image views may not have been intentional, such as people pressing the previous button to go back to an earlier image, as described in the next section.)

During that period, file page views declined by 73%, from about 12 million in May, before Media Viewer was launched on English and German Wikipedias, down to about 3 million by the end of 2014.

Note that these metrics do not include the variance from earlier releases on other large wikis like the French or Spanish Wikipedias and Wikimedia Commons, as we did not collect all these metrics prior to May 2014.

So the actual variance is likely to be higher, but would require more research to determine.

Intentional Views
Not all Media Viewer image views can be considered 'intentional' -- here's one way to separate them from each other:

'Intentional' views 	(~17M/day)
 * thumbnail + magnifier clicks
 * next clicks
 * hash loads
 * file page views

'Unintentional' views 	(~6M/day)
 * previous clicks
 * history navigation events

On that basis, 'intentional' image views can be estimated at about 17M (versus ~6M for 'unintentional' views).

Overall, 'intentional' views increased significantly, growing by about 50% after the release on English and German Wikipedias, when we started tracking these metrics.

Note that this does not include the growth from the earlier releases on other large wikis like the French or Spanish Wikipedias and Wikimedia Commons.

So the actual growth is likely to be higher, but would require more research to determine.

Enabled Rates
''Percent of global logged-in users with Media Viewer enabled or disabled as of Dec. 8, 2014, by user group. Source: WMF.''

Media Viewer was enabled for about 99.54% of global logged-in visitors to Wikimedia sites as of December 8, 2014 (versus 0.46% disabled).

Those rates are lower for advanced users:
 * 95.57% of active contributors had Media Viewer enabled (users with 5+ edits / month)
 * 85.91% of very active contributors had Media Viewer enabled (users w/ 100+ edits / month)

These numbers were collected a month after a very prominent enable/disable button was added to Media Viewer, to make it easier for users who didn't want the tool to opt-out.

A month later, in January 2014, these percentages had not changed significantly, suggesting that 	that enabled rates are stabilizing.

Disable Rates - Cumulative
Cumulative opt-out rates for logged-in users show more active users disabled Media Viewer from Aug. to Dec. 2014. [http://multimedia-metrics.wmflabs.org/dashboards/mmv#opt_in_and_opt_out-graphs-tab Source: WMF

This cumulative chart of global disable rates by logged-in users shows that about 0.49% had opted-out by the end of December 2014, compared to total visitors who visited in the previous 30 days.

Those disable rates are higher for advanced users, with 4.88% for active contributors (5 edits per month) and 15.51% for very active contributors (100+ edits per month).

Overall, disable rates seem to be stabilizing across all logged-in user groups.

Opt-ins/outs - Logged-in vs. anonymous
''Global opt-out events by anonymous users started in July 2014 and varied as new features were released, with a downward trend by year-end. Source: WMF''

Anonymous opt-outs and opt-ins increased in early September, after File Usage was removed, making the Disable/Enable links easier to find.

After the Enlarge feature was released in late September, global anonymous opt-outs dropped by about 60%, from 2.2k opt-outs per day down to 1k. The Enlarge feature addresses one of the most frequent user requests, enabling users to zoom in on images.

After a more prominent Disable/Enable button was released in November, anonymous opt-outs events nearly tripled, from 1k back up to 3k opt-outs per day -- and opt-outs by logged-in users also increased to an average of 100 opt-outs/day.

By the end of 2014, opt-outs for both anonymous and logged-in users had dropped by half again, from a peak of 3.6k down to 1.9k total opt-outs per day.

Opt-outs by logged-in users surged briefly in late August, after the default for Wikimedia Commons was switched to opt-in for logged-in users, to facilitate curation on that multimedia file repository.

Other feature changes did not seem to impact opt-outs by logged-in users, which were steady at about 65 opt-outs/day by year-end.

Opt-ins by anonymous users seemed less impacted by feature changes, representing about one-sixth of opt-outs by the end of the year. (Note that opt-ins by logged-in users is not shown, due to a bug).

Note that for technical reasons, it is quite hard for anonymous users to re-enable Media Viewer on most wikis, unless they do so right after disabling. So the opt-in events on this chart may be due to users trying out the disable feature and choosing to re-enable it.

Activity Metrics
Here are some of the user actions tracked through EventLogging in 2014. Please refer to the image in the Features section above to see how these actions are shown in Media Viewer's final user interface.

Top 5 Actions


Here are the top 5 actions for Media Viewer, tracked by number of daily clicks:


 * Open/Thumbnail clicks (10.3M)
 * Close (9.1M)
 * Next (6.8M)
 * Previous (2.5M)
 * History (4M)

Every day, we log an average of 10 million clicks on thumbnails in articles -- and almost as many clicks on the Next and Previous buttons. This new browsing feature did not exist before Media Viewer. Its widespread use suggests that Media Viewer may be adding value for millions of readers, who might not be using this browsing feature if they did not find it useful.

History tracks how often users navigate within Media Viewer using their browser's history.

Enlarge & Full Screen


To address frequent requests for a 'zoom' function, a new 'Enlarge' feature lets you click on an image to view its original file in full resolution: it is now used 12x more often (1M/days) than the full screen feature (93k/day), as of December 7, 2014. We also observed a dramatic decline in opt-out rates after this new feature was introduced (as shown in a separate graph).

This data and our design research suggest that the Enlarge feature significantly improved overall user experience by enabling users to view images in detail. In usability studies, the full screen feature is still found useful, because it lets you fill the screen with the selected image -- but not as useful as zooming onto the image.

Metadata & Details


This chart compares global daily events for opening (or closing) the metadata panel, versus clicking on 'more details' to go to file description pages: the metadata panel is now used 400x more often (1.3M/day) than the 'More details' button (~67k) in Media Viewer. Note that this total includes scrolling events to open the metadata panel (1.2M/day), as well as keyboard presses on the up arrows (130k).

It's also worth noting that the metadata panel open frequency is around 7% of image view frequency, so people actually use the panel fairly often (way more than the 'More details' button to access the file description page, which is ~0.25%).

Overall, this data suggests that the metadata panel is valuable to users.

Download, Disable & Share


The Download panel is opened more often (74k / day) than the Disable/Enable panel (26k / day -- labeled 'options' in this graph) or the Share/Embed panel (20k / day -- labeled 'use this file' in this graph).

Information Links


In the information that appears below the image, users click more often on Source links (7k/day) than on Author links (3k/day).

They also click a bit more on 'View terms' (4k/day) than on License links (3k/day - e.g. 'CC-BY-SA').

Predictably, links that are below the fold (e.g. Uploader, About) get fewer clicks than above the fold, because users have to scroll down or click on an ellipsis to open the metadata panel that contains them. It's also worth noting that clicks on source and terms decreased by half after we launched the captions above the fold, perhaps because that new information in the first two lines detracted from those links on the third line.

New Features
''Global clicks on 'Enlarge', 'More details' and 'Download' surged after launch in Sep.-Oct. 2014. Source: WMF''

After the new Enlarge feature was released, we observed a dramatic 20x increase in clicks for this popular 'zoom' feature, one of the most frequent user requests. By clicking anywhere on the image, users can now go to the original file and use their browser's zoom tool to enlarge image details.

The release of the the big blue 'More details' button also registered a 3x increase in clicks, compared to the Commons icon that was featured before (both link to the file description page).

And giving the Download feature its own icon increased clicks by 3x as well (it used to be a tab in a single 'Use this file' panel, along with 'Share' and 'Embed').

Media Viewer vs. File Page
''Global image load times for Media Viewer vs. File Pages are on par, for either median or 95th percentile, Dec. 2014. Source: WMF''

This chart compares image load times between Media Viewer and the File page in Dec. 2014.

It suggests that load times are about the same, with a median load time of 1.5 seconds to 2 seconds -- even though Media Viewer typically loads larger images than the File Page.

For the 95th percentile (the longest average load for 95% of users), both viewing methods are also on par, with load times in the 8-10 seconds range. Note that File Page performance improved significantly after WMF implemented a switch to HHVM in November 2014: prior to that, Media Viewer was loading images about twice as fast as the file page.

Network Performance: API


This chart shows the average duration of image loads per country, in milliseconds, as a reflection of network performance. Faster load times are shown in green, slower load times in red.

Overall network performance seems related to Internet bandwith in each region, with faster load times in developed countries than in the developing world.

Attributions & Licenses


A growing number of images shown in Media Viewer now have machine-readable data for author, source or license information.

These ratios have been increasing steadily, as a percent of image views: 97% now have license information, 86% have sources and 86% have authors.

These improvements are related to a recent drive to clean up metadata for multimedia files across Wikimedia sites.

Design Research
The Wikimedia Foundation's design research team conducted a range of design research studies and usability tests for this project:


 * Study 1 (July 2014)
 * Study 2 (August 2014)
 * Study 3 (September 2014)
 * Study 4 (November 2014)
 * Usability tests (Feb.-May 2014)

User Study 1
Here are key issues from our first user study of the current version in July:
 * Hard to discover info panel
 * Info panel covers images & is hard to control
 * Hard to find 'View original file' button
 * Hard to find 'Download' button
 * Clarify meaning of viewing options

See User study 1 slides (July 2014).

User Studies 2 and 3
Users successfully completed all tasks listed below:

Primary Tasks
 * Preview images
 * Learn about images
 * Get more details
 * Browse images
 * Disable feature (passed 2nd test)
 * Re-enable feature (passed 2nd test)

Secondary Tasks
 * Enlarge
 * Share
 * Download
 * Go back

See User Study 2 Slides (August-September 2014).

User Study 4
We observed that readers were able to accomplish these key tasks with no problems:
 * View images in Media Viewer
 * Navigate to next / previous image
 * Read a long description
 * Identify details about images
 * Get more details on file pages
 * Zoom in to see details
 * View image in full screen
 * Share an image with a friend
 * Disable Media Viewer
 * Re-enable Media Viewer

See User study 4 report (November 2014).

Feedback channels
User feedback came from a variety of channels:
 * 12+ discussions (in-person, IRC, hangouts)
 * 20+ talk pages (mw.org, dewiki, enwiki, frwiki, etc.)
 * 25K+ beta testers (for 6 months, throughout development)
 * 18K+ survey responses (throughout release, in 8 languages)
 * 30+ user tests and studies (throughout development and release)
 * 3 requests for comments (RfCs on enwiki, drwiki + commons)
 * 1 community consultation (on meta.wikimedia)

Talk Pages
Here are some key issues reported in talk pages after release:
 * Make it opt-in
 * Hard to find file page link
 * Show caption higher up
 * Can't go back quickly
 * Image load too slow
 * Issues on tablets/mobile
 * Show full file name
 * Show more license info
 * Show label for author
 * Make image clickable

See Media Viewer Talk Page.

Requests for comments (RfCs)
Three community RfCs were held from June to October 2014:
 * English Wikipedia RfC
 * German Wikipedia RfC
 * Commons RfC

RfCs shared these attributes:
 * low representation for readers, our target users
 * low participation by a few hundred self-selected contributors
 * low signal-to-noise ratio, with few actionable suggestions for improvement
 * all RfCs requested that Media Viewer be disabled by default

WMF committed to more improvements, based on community consultation (see next section).

Community Consultation
A community consultation was held on Meta, from August 28 to September 7:
 * the goal was identify any critical issues
 * it was widely promoted on community channels
 * community members made 130 suggestions
 * WMF evaluated and prioritized all suggestions
 * WMF selected 10 'must-have' tasks
 * WMF developed 13 new improvements
 * all improvements were released and validated by Nov. 2014
 * see improvements list below

See Consultation Page.

Improvements
These improvements were made from Sep. to Nov. 2014, based on community consultation:
 * An easier way for any user to disable the tool for their personal use
 * Enlarge images by clicking on them
 * A caption or description right below the image
 * A more prominent link to the File: page: "More Details" button
 * Separate icon for Download
 * Clearer icon for Share / Embed
 * simpler metadata panel with fewer items
 * Show file page link if attribution is missing
 * Fix 'Licensed under Fair Use' credits
 * Support metadata cleanup drive

See Improvements Page.

Surveys
Users were asked for feedback in a short survey (April-July 2014):
 * optional survey (subject to self-selection bias)
 * built-in feedback button (opened survey popup window)
 * 8 different languages (ca, de, en, es, fr, hu, nl, pr, pt)
 * 18K+ survey responses (satisfaction ratings, some with comments)
 * a majority found the tool useful (but sample may not be representative)
 * a lot of actionable feedback (hand-coded to identify and prioritize key issues)

See Survey report.

Survey feedback
Here are top feature requests from surveys: (% hand-coded survey requests)
 * Want zoom 	14%
 * More image sizes 	8%
 * Want opt-out 	6%
 * Easy way to exit 	5%
 * Scroll down for info 	5%
 * Info cuts off images 	5%
 * Can't find info 	4%
 * Show whole image 	3%
 * More info above 	3%
 * Disoriented by UI 	3%

See Survey feedback.

Survey results


Here are the final survey results, based on 18,199 global responses as of July 8, 2014, when all surveys ended.

Note that this was an optional survey, so approval rates should not be cited as a conclusive metric, as they are subject to self-selection bias. The primary purpose of the survey was to collect user comments in order to improve the tool -- not to use approval ratings as a definitive measure of success. Also, this global chart adds together feedback from different sites with different response rates and survey running times.

Keeping these caveats in mind, a majority of global respondents found the tool "useful for viewing images and learning about them" (56% "useful", 35% "not useful", 9% "not sure"), based on total unweighted survey results, as shown on this spreadsheet.

When weighted to match global Wikipedia readership for each language, fewer respondents found the tool "useful" than "not useful" (39% "useful", 50% "not useful", and 10% "not sure").

See Survey results.

Approval rates


On average, about 65% of readers found Media Viewer useful, based on 7,582 reader survey responses in 7 languages, over a 12-week period, from 14 April to 7 July, 2014 (see linked spreadsheet).

Weekly reader approval rates are shown here for each language. The drop in non-English rates in May-June may be due to non-English readers of Enwiki and Commons: the surveys are based on interface language, not content language.

Cumulative "useful" approval ratings from all users varied greatly by language: English 36%, French 70%, Spanish 78%, Dutch 59%, Portuguese 81%, German 30%, Hungarian 63%, Catalan 71%.

Overall approval rates thumb|240px|page=41|from English users gradually increased in the month following launch: the weekly percentage of survey respondents who found Media Viewer useful ("yes") grew from 26% in May to 52% in June 2014. Note that the survey response rate fell at the same time, as shown in these results.

Again, note that this was an optional survey, so approval rates should not be cited as a conclusive metric, as they are subject to self-selection bias. They are only shown here for the record, as they were frequently referred to in community discussions.

Related documents

 * Research Slides
 * Research Spreadsheet
 * Live Metrics Dashboards
 * More Metrics Dashboards
 * Project Timeline
 * About Media Viewer
 * Media Viewer Demo

Thanks
This research was collected and analyzed for over a year by many different individuals, who worked diligently, above and beyond the call of duty, to help the development team make more informed decisions. We are very grateful to everyone who made this possible!

Research contributors include: Aaron Arcos, Daisy Chen, Gilles Dubuc, Fabrice Florin, Pau Giner, Mark Holmquist, Keegan Peterzell, Abbey Ripstra, Gergo Tisza.

Research advisors include: Howie Fung, Aaron Halfaker, Rob Lanphier, Erik Moeller, Toby Negrin, Dario Taraborelli, Lila Tretikov.

This report was prepared by Fabrice Florin and Gergo Tisza, on behalf of the team, from Dec. 2014 to Jan. 2015.