Reading/Web/Desktop Improvements/Repository/Sticky Header and Table of Contents User Testing

From mediawiki.org
The full report of findings from the study

The sticky header and table of contents (ToC) improvements are proposed changes as a part of the desktop improvements project. These would provide access for readers and editors to navigation and other important functionality throughout the page.

These changes were inspired by data coming from various sources. We have been receiving requests from editors. Also, prior to the beginning of the project, we performed generative research which brought similar conclusions.

We have committed to testing our prototypes and ideas with both readers and editors in a variety of different settings. For the sticky header and table of contents, we began this testing by preparing prototypes for a few different versions of the feature. Next, we tested it with readers and editors in three emerging regions - Ghana, Indonesia, and Argentina.

Our goal was to determine the overall utility of each prototype as well as to gain insight into the changes we would need to make for the future. This report shows the findings of this testing and highlights the next steps for the features themselves.

Introduction

Currently, many useful tools and functions are only available at the top of the page, and/or on the left panel of the page. This includes search and language switching, and navigation (e.g. ToC). We want to experiment with increasing access to popular tools or functions by making them available from any point on the page.

Through this research, we explored two possible ways of doing this.

Sticky header

On longer article pages, scrolling past the first few paragraphs means the user would need to scroll back up to access the tools and other resources again. Our proposed method of addressing this is to make the site header (and/or the article header) “sticky”. This means the header stays fixed to the top of the screen (above the content) as you scroll down. Given that space on a hypothetical sticky header (that would persist as the user scrolls down through article pages) would be limited, research was needed to determine what functionalities and tools would be most useful to include.

Our research in this area focused on answering the question: Assuming we have limited space for persistent tools and functions, which ones would be most valuable to readers, and (separately) to editors?

Table of contents

The table of contents is crucial to the in-page navigation experience. Previous research has shown that users look to it to understand the article structure and contents, and allow for quick navigation to said contents. Currently, most articles have a table of contents near the top of the page.

Our research in this area focused on determining whether a table of contents that is constantly accessible (instead of being just a static element near the top of a page) while browsing an article would provide more value to users than the current setup. Subsequently, we also wanted to determine the best possible way of providing this functionality.

Methods

This research was generated and coordinated by the Design Research team at the Wikimedia Foundation. The prototypes shown to readers and editors were created by the Web team. The research was conducted by three external vendors:

  • URIKA Insights in Ghana
  • Terang Riset Pratama in indonesia
  • CRIBA Research in Argentina

Participants

Vendors in three emerging regions (Ghana, Indonesia, Argentina) were engaged to evaluate table of contents (ToC) prototypes and sticky header (SH) elements/usability. The research was conducted in three languages - English (Ghana), Bahasa Indonesia (Indonesia), and Spanish (Argentina). Each testing location had 6 participants that consisted of approximately ½ newcomers or casual readers, and ½ regular editors on their respective language wikis.

Sticky header

For the sticky header, two prototypes were created that showed a possible implementation of the sticky header - one reader-focused prototype and one editor-focused prototype. Participants were asked to interact with the prototypes and rate the usefulness of the functionality within the prototypes for their daily usage of Wikipedia

Research Goals:

  • Determine ideal elements to include for readers (i.e. logged-out users)
    • Search, article title, language switching, etc.
  • Determine ideal elements to include for editors (i.e. logged-in users)
    • Talk, History, language switching, search, etc.
      • Determine if this list would be different for this cohort if they weren’t logged in
  • Establish relative priority for the elements for each cohort
  • Evaluate prototypes reflecting the elements determined above by cohort

Research Questions:

  • Assuming we have limited space for persistent tools and functions (potential customization available for logged in in the future, not planned for logged out), which of the following would be most valuable:
    • Logged-out
      • Wikipedia logo
      • Article title
      • Section title
      • Search
      • Language switching
      • Edit
    • Logged-in
      • Wikipedia logo
      • Article title
      • Section title
      • Search
      • Language switching
      • Edit/edit source
      • Talk
      • History
      • Watch
      • Page tools (e.g. move)
      • User tools (opens up a menu that have to do with the user: user page, sandbox, preferences, talk beta watchlist contributions, alerts, notifications)
      • Your favorite gadgets (e.g. twinkle, tool from preferences)
    • Which elements should be persistent?
  • Which elements should be surfaced upon a ‘need’ trigger (scroll-up, hover, other?)

Table of contents

For the table of contents, participants were first asked to explore the site naturally. A general idea of their opinion on the usage of the table of contents was established. Then, a variety of prototypes were provided for different potential versions of the table of contents. The focus here was on presenting different ways in which the ToC could be accessed from any place within the page. Participants were asked to rank their preferred prototype, as well as to give feedback on the overall idea of a table of contents available throughout the page.

Research Goals:

  • Establish a qualitative understanding of how the table of contents is used, i.e. the ways in which it provides value to readers
    • How does this vary on very short pages, medium length pages, and long pages
  • Establish an understanding of other methods of in-page navigation are being used (e.g. find in page)
  • Establish an understanding of what value, if any, having access to the table of contents (or a similar navigational tool) from anywhere in the page would offer
    • Is there a separate value in just seeing where you are and what the lay of the land is versus from actually clicking and navigating with it?
  • Evaluate 3-4 different in-page navigation / persistent table of contents solutions (in the form of prototypes)
    • 1) persistent table of contents
    • 2) table of contents via a menu in the article header
    • 3) table of contents via a drawer next to the article

Research Questions:

  • Could the table of contents provide more value to readers if it was accessible from anywhere in the article (rather than just at the top)?
  • What are the main use cases of the ToC? How do people think of it?
    • If people interact with the ToC do they typically click on one link, or more than one?
    • How do people use the ToC to navigate the article?
    • What are the main shortcomings, if any, of the current ToC?
  • Does the usage follow any sort of pattern (e.g. long vs. short articles, new vs. experienced users)?
  • What are the main differences (or tradeoffs) users notice between option A (new ToC sidebar) and option B (supplemental ToC)?
  • How does one perceive the contents of an article with/without a persistent TOC?
  • Do we lose legibility / scannability with persistent because it is more narrow?
  • Lower priority questions:
    • Right vs left?
    • How many levels of depth?
    • Numbers vs. no numbers?

Results

Overall

Both the sticky header and table of contents prototypes were received well by all groups tested across languages.

For the sticky header, a number of suggestions were presented and a prioritization of most-useful elements was provided. The prioritization of elements varied by location and logged-in/logged-out status (more details below)

Sticky header

We asked all testers (readers and editors) to prioritize the list of functionality that would be most useful to them to have access to at all times. Here, the results varied between languages, but were not significantly different. Below, we are providing the combined prioritization (the location-based prioritization lists are available in the full report above).

Logged-out users:

  1. Article Title
  2. Section Title
  3. Search
  4. Language Switching
  5. Wikipedia logo
  6. Edit

Logged-in users:

  1. Search
  2. Article Title
  3. Talk
  4. Edit
  5. Language Switching
  6. History
  7. User Tools
  8. Page Tools
  9. Watch
  10. Gadgets
  11. Wikipedia Logo
The proposed sticky header prototype for logged-in users

We also asked participants to review the prototypes available for logged-in and logged-out users. All users reported positive experiences with the prototypes. Some users requested that the sticky header is always available rather than triggered upon scrolling. Editor participants frequently requested further customization of the sticky header elements. When asked to determine the legibility of the iconography used in the sticky header, many users raised questions on the meanings of several of the icons used in the prototypes, in particular the icon for beta features, gadgets, and contributions.

Table of contents

All users found the table of contents to be essential to the reading experience of Wikipedia. The results also gave insight into the primary use cases for the use of the table of contents:

  • Getting a quick sense of the contents of the page - what information is available on the page/is the information the participant is looking for available on the page?
  • Getting a sense of the shape/map of the page and creating a mental model - how is the page organized, which sections are longer than others
  • Direct navigation to a specific section or piece of information

Behavior change based on user type

These results varied slightly depending on whether the participant was a newcomer, casual reader, or editor, with casual readers and editors being slightly more likely to interact with the table of contents directly prior to interacting with any other part of the article on medium and long pages. On short pages, the table of contents was used less-frequently. All users indicated that having the ToC only at the top of the article was an issue.

Newcomers:

  • Tend to scroll around the article to look for a topic
  • ToC is used a context more than navigation

Casual Readers:

  • Tend to go directly to the ToC - it was their first stop to understanding the context of the article.
  • Once context is established, casual readers tended to use the ToC more for navigation than  newcomers

Editors:

  • Go directly to the ToC
  • Used the ToC as the main point for navigation (rather than scrolling around)
  • Tended to return to the ToC while reading the article for reference or navigation

The table of contents prototypes ranking

Participants were asked to interact with five different prototypes of the table of contents. The persistent prototype was the highest ranking of all the available prototypes across all groups. No significant differences were observed based on location or familiarity with Wikimedia wikis (newcomers, casual readers, editors.)

The remainder of the rankings were as follows.

  1. Persistent. This prototype displayed the ToC in the left margin of the page. It also included the ability to display all sections and subsections using collapsible arrows on the side of the prototype. Participants indicated that constant access to the ToC was the main reason for the selection of this prototype as it did not require any further clicks or other interaction.
    A prototype of the persistent table of contents for Wikimedia wikis created for the purposes of user testing
  2. ToC in Header. This prototype presented the ToC within a sticky header on the page. Participants indicated that the ease of use of this prototype was high and particularly noted satisfaction with the clarity of which section and subsection the article was on. However, participants were unsatisfied with the collapsible nature of the prototype and requested for persistence.
    A prototype of a header-based table of contents for Wikimedia wikis created for the purposes of user testing
  3. Floating dots prototype. This prototype presented the ToC as a persistent but collapsed element on the side of the page that was triggered on hover. Once again, the persistent element of the ToC was rated highly. However, participants indicated that they did not like the ToC overlapping the article text when opened.
    A prototype of a floating box table of contents for Wikimedia wikis created for the purposes of user testing
  4. Floating box. This prototype presented a small box in the right bottom corner of the page, which could be expanded into a persistent element. During testing, participants had difficulty noticing the element. When expanded, they also noted dissatisfaction with the ToC overlapping the article text
    A prototype of a dots-based table of contents for Wikimedia wikis created for the purposes of user testing
  5. Back to top button. This prototype presented a button that took participants to the top of the page, where the original ToC is presented. This prototype was not received well across all groups. Participants mentioned that it did not provide any additional functionality that cannot be covered by the browser or the home button on their keyboards.
    A prototype of a back to top button on Wikipedia

Other crucial observations:

  • Participants prioritized persistent access - the best-performing prototype was the persistent one across all tests
  • Participants prioritized having more information - prototypes that contained all sections and subsections performed better
  • Participants did not want the ToC to overlap the content, even in the cases where it was supplementary to the main toc at the top of the page
  • Participants liked getting a sense of location within the article and noted that additions like bolding the title or the section helped with their orientation

Discussion and next steps

This study provided crucial information for the development of the sticky header and table of contents proposed features. As both features were received well overall by participants, the web team will continue with the development of the features and make changes according to the results mentioned above.

Sticky header

We iterate on the prototypes available, focusing on adding the items prioritized as most important to the following sticky header prototypes. We will also explore different triggering mechanisms for the sticky header, including allowing the header to be persistent across the top of the page.

We will be continuing this research into the sticky header by running wider testing of our prototypes with editors and other logged-in users across a variety of different Wikimedia projects.

Table of contents

As the persistent prototype was ranked highest in all groups, we will consider iterations to this prototype only for future planning. We will also explore collapsible options for situations when presenting the full list might be difficult or inefficient. For example, this may be smaller screen resolutions.

We will be continuing this research into the table of contents by running wider testing of our prototypes with editors and other logged-in users across a variety of different Wikimedia projects.