Wikimedia Research/Design Research/Reading Team UX Research/Hovercards Usability

Hovercards Usability Study

By Sherah Smith and Daisy Chen = Abstract = This user research looks at the usability of the Hovercards feature on Desktop English Wikipedia to determine the usefulness of Hovercards, understand any usability issues around the feature, and explore user attitudes towards the experience overall. Through recorded video testing and written surveys, we determined that Hovercards were mostly received positively, and that they met the goal to assist readers with learning general information about links that they hovered over during their visit to Wikipedia. The data we collected during this study is useful for iterating the Hovercards feature and for thinking about more ways to test its usefulness in other contexts.

= Background = The Hovercards beta feature potentially solves the core problem of users needing to open multiple tabs to gain an understanding of a word in the context of the subject they are reading. With Hovercards, whenever a reader hovers over a link to another article, a short summary of the subject and an image (if available) is shown to them, so that they can decide whether they need to visit that subject more fully before continuing the current subject.

= Research Questions =
 * What do users feel about Hovercards?
 * How do users feel about the workflow for turning Hovercards on or off?
 * Are Hovercards discoverable?
 * Are they distracting, or a nuisance?
 * What are users calling the Hovercard?

= Methodology =
 * Unmoderated remote panel study using our new software for such studies, UserZoom. Participants’ screens and faces are recorded, with permission, as they follow instructions for 5 tasks. They finish by filling out a survey.

= Participants = Round 1:

= Summary of Test and Findings =

Questions

 * 1) Search for “Cactus” article. Read through as you normally would read any Wikipedia article.
 * 2) Log in to Wikipedia, enable all Beta features.
 * 3) Go to Wikipedia main page and place the cursor over the first link under the “From Today’s Featured Article” section. Say what you think about what happens.
 * 4) Go to “Ghost” article and read through as you normally would read any Wikipedia article.
 * 5) Without going to the “Beta” section, show how you would disable Hovercards.

Observations: Attitudes

 * 13 out of 15 questionnaire participants reported positive experiences.
 * 3 of 5 video participants had immediate, positive reactions. The other two acted as though they had experienced expected behavior (the Hovercard seems natural, not noteworthy or surprising).

Observations: Video Participant Behaviors

 * 4 of 5 read articles by following words with their mouse.
 * About half of the participants do an “overview glance,” using TOC or article skimming, before reading an article outright.
 * Only one person actually clicked through to a page that was previewed in a Hovercard.
 * 4 of 5 hovered on several (3+) links in articles after being introduced to Hovercards. This was mostly for definitions.
 * 3 people follow links more before being introduced to Hovercards.
 * 3 people love the pictures on Hovercards. One very decidedly does not like them.

Answers to Research Questions

 * 1) What do users feel about Hovercards ? Generally (13/15) they approve. A+
 * 2) How do users feel about the workflow for turning Hovercards on or off? Only one user complained. A+
 * 3) Are Hovercards discoverable? Yes.
 * 4) Are they distracting, or a nuisance? No.
 * 5) What are users calling the Hovercard?

= Screener = Round 2:

= Protocol = Task 1 of 5: To start this test, search for the article for "Cactus" on Wikipedia without logging in. Take some time to read or skim the article. Go at whatever pace you like, but make sure to read all the way to the bottom of the article. Just read through as you would normally do. If you feel like going to other pages, that’s ok, but come back to this one when you’re finished.

Task 2 of 5: Now, log in to Wikipedia. Once you are logged in, enable beta features by following the "Beta" link in the upper-right corner of the screen. Check each checkbox under the "beta features" tab you see, save the page, and then continue.

Task 3 of 5: Return to https://en.wikipedia.org/wiki/Main_Page, and in the section that says “From today’s featured article”, place the cursor over the first link in the text (but don’t click). What happens when you do place the mouse over the link? What do you think about this?

Task 4 of 5: Now, navigate to the article for "Ghost" (from folklore and mythology). Take some time to read or skim the article. Go at whatever pace you like, but make sure to read all the way to the bottom of the article. Just read through as you would normally do. If you feel like going to other pages, that’s ok, but go back to "Ghost" when you’re finished.

Task 5 of 5: Without going to the "beta" section of this page, can you show how you would disable the feature that you see appearing when you place your mouse over links?

Final Questionnaire
1. How was your experience using Wikipedia today?

2. What would you call the thing that appears when you place your cursor over a link?

3. Do you have any feedback about what you saw appear when you held your cursor over a link? 4. Which variation of this feature do you prefer? Rank in order of most favorite to least favorite.

Round 1: Round 2:

5. Why did you rank the variations from the previous question the way that you did?

6. Do you have any other feedback?