Hovercards Usability Study

By Sherah Smith and Daisy Chen

Abstract[edit]

This user research looks at the usability of the Hovercards feature on Desktop English Wikipedia to determine the usefulness of Hovercards, understand any usability issues around the feature, and explore user attitudes towards the experience overall. Through recorded video testing and written surveys, we determined that Hovercards were mostly received positively, and that they met the goal to assist readers with learning general information about links that they hovered over during their visit to Wikipedia. The data we collected during this study is useful for iterating the Hovercards feature and for thinking about more ways to test its usefulness in other contexts.

Background[edit]

The Hovercards beta feature potentially solves the core problem of users needing to open multiple tabs to gain an understanding of a word in the context of the subject they are reading. With Hovercards, whenever a reader hovers over a link to another article, a short summary of the subject and an image (if available) is shown to them, so that they can decide whether they need to visit that subject more fully before continuing the current subject.

Research Questions[edit]

What do users feel about Hovercards?
How do users feel about the workflow for turning Hovercards on or off?
Are Hovercards discoverable?
Are they distracting, or a nuisance?
What are users calling the Hovercard?

Methodology[edit]

Unmoderated remote panel study using our new software for such studies, UserZoom. Participants’ screens and faces are recorded, with permission, as they follow instructions for 5 tasks. They finish by filling out a survey.

Participants[edit]

1. Participant profile

Answer	Count	Percent
UZ recruited - Male	4	57%
UZ recruited - Female	3	43%
answered question	7
skipped question	0

2. Operating System

Answer	Count	Percent
Mac OS	3	43%
Windows 10	4	57%
answered question	7
skipped question	0

3. Browsers

Answer	Count	Percent
Chrome	7	100%
answered question	7
skipped question	0

Round 1:

1. Participant profile

Answer	Count	Percent
UZ recruited - Male	5	63%
UZ recruited - Female	3	38%
answered question	8
skipped question	0

2. Operating System

Answer	Count	Percent
Windows 7	1	13%
Mac OS	2	25%
Windows 10	5	63%
answered question	8
skipped question	0

3. Browsers

Answer	Count	Percent
Chrome	8	100%
answered question	8
skipped question	0

Summary of Test and Findings[edit]

Questions[edit]

Search for “Cactus” article. Read through as you normally would read any Wikipedia article.
Log in to Wikipedia, enable all Beta features.
Go to Wikipedia main page and place the cursor over the first link under the “From Today’s Featured Article” section. Say what you think about what happens.
Go to “Ghost” article and read through as you normally would read any Wikipedia article.
Without going to the “Beta” section, show how you would disable Hovercards.

Observations: Attitudes[edit]

13 out of 15 questionnaire participants reported positive experiences.
3 of 5 video participants had immediate, positive reactions. The other two acted as though they had experienced expected behavior (the Hovercard seems natural, not noteworthy or surprising).

Observations: Video Participant Behaviors[edit]

4 of 5 read articles by following words with their mouse.
About half of the participants do an “overview glance,” using TOC or article skimming, before reading an article outright.
Only one person actually clicked through to a page that was previewed in a Hovercard.
4 of 5 hovered on several (3+) links in articles after being introduced to Hovercards. This was mostly for definitions.
3 people follow links more before being introduced to Hovercards.
3 people love the pictures on Hovercards. One very decidedly does not like them.

Answers to Research Questions[edit]

What do users feel about Hovercards ? Generally (13/15) they approve. A+
How do users feel about the workflow for turning Hovercards on or off? Only one user complained. A+
Are Hovercards discoverable? Yes.
Are they distracting, or a nuisance? No.
What are users calling the Hovercard?

Page Preview	3
Information Box	2
Pop-up	2
Information card
Tooltip
Bubble
Index Box
Preview Plane
Hover Hint
Hyper Links
Information Window

Screener[edit]

1. What is your gender?

Answer	Count	Percent
Female	3	38%
Male	5	63%
answered question	8
skipped question	0

2. Please select your age group:

Answer	Count	Percent
Under 18	0	0%
18-25	1	13%
26-35	1	13%
36-45	0	0%
46-55	6	75%
56-65	0	0%
66+	0	0%
answered question	8
skipped question	0

3. To complete this study, you will need to use a desktop computer with a mouse. It will not work with mobile or touch devices. Are you using a desktop computer with a mouse?

Answer	Count	Percent
Yes	8	100%
No	0	0%
answered question	8
skipped question	0

4. Do you currently have an account with Wikipedia that you are able to log in to?

Answer	Count	Percent
Yes	8	100%
No	0	0%
answered question	8
skipped question	0

5. You will be required to "think out loud" as you complete this study. When you "think out loud", you can talk about your reactions to what you are seeing, doing or feeling as you go through each task and question.Are you willing to "think out loud" while giving your feedback?

Answer	Count	Percent
Yes	8	100%
No	0	0%
answered question	8
skipped question	0

6. Please ensure you clicked "ALLOW" on your settings to record your verbal feedback on your desktop.Please check all the requirements below before proceeding. [Check all that apply]

Answer	Count	Percent
Your location/room is well-lit	8	100%
You are in a quiet environment with no background noise	8	100%
Audio is not muted and volume is turned up to highest level	8	100%
Your camera & microphone are operational (as far as you are aware)	8	100%
answered question	8
skipped question	0

Round 2:

1. What is your gender?

Answer	Count	Percent
Female	3	43%
Male	4	57%
answered question	7
skipped question	0

2. Please select your age group:

Answer	Count	Percent
Under 18	0	0%
18-25	0	0%
26-35	5	71%
36-45	1	14%
46-55	0	0%
56-65	1	14%
66+	0	0%
answered question	7
skipped question	0

3. To complete this study, you will need to use a desktop computer with a mouse. It will not work with mobile or touch devices. Are you using a desktop computer with a mouse?

Answer	Count	Percent
Yes	7	100%
No	0	0%
answered question	7
skipped question	0

4. Do you currently have an account with Wikipedia that you are able to log in to?

Answer	Count	Percent
Yes	7	100%
No	0	0%
answered question	7
skipped question	0

5. You will be required to "think out loud" as you complete this study. When you "think out loud", you can talk about your reactions to what you are seeing, doing or feeling as you go through each task and question.Are you willing to "think out loud" while giving your feedback?

Answer	Count	Percent
Yes	7	100%
No	0	0%
answered question	7
skipped question	0

6. Please ensure you clicked "ALLOW" on your settings to record your verbal feedback on your desktop.Please check all the requirements below before proceeding. [Check all that apply]

Answer	Count	Percent
Your location/room is well-lit	7	100%
You are in a quiet environment with no background noise	7	100%
Audio is not muted and volume is turned up to highest level	7	100%
Your camera & microphone are operational (as far as you are aware)	7	100%
answered question	7
skipped question	0

Protocol[edit]

Task 1 of 5: To start this test, search for the article for "Cactus" on Wikipedia without logging in. Take some time to read or skim the article. Go at whatever pace you like, but make sure to read all the way to the bottom of the article. Just read through as you would normally do. If you feel like going to other pages, that’s ok, but come back to this one when you’re finished.

Task 2 of 5: Now, log in to Wikipedia. Once you are logged in, enable beta features by following the "Beta" link in the upper-right corner of the screen. Check each checkbox under the "beta features" tab you see, save the page, and then continue.

Task 3 of 5: Return to https://en.wikipedia.org/wiki/Main_Page, and in the section that says “From today’s featured article”, place the cursor over the first link in the text (but don’t click). What happens when you do place the mouse over the link? What do you think about this?

Task 4 of 5: Now, navigate to the article for "Ghost" (from folklore and mythology). Take some time to read or skim the article. Go at whatever pace you like, but make sure to read all the way to the bottom of the article. Just read through as you would normally do. If you feel like going to other pages, that’s ok, but go back to "Ghost" when you’re finished.

Task 5 of 5: Without going to the "beta" section of this page, can you show how you would disable the feature that you see appearing when you place your mouse over links?

Final Questionnaire[edit]

1. How was your experience using Wikipedia today?

Answer
My experience was easy and fun. I also learned about some features I did not know about!
It was good! I like the beta features. Helps give details without leaving the main webpage you are on
I did not understand how to turn off the hovercards. I felt that it was very confusing to navigate. The rest of my experience was as expected and how I would typically use the website.
Fantastic! I never bothered with any beta features in the past, probably because I am usually not logged in. But, I really liked the preview feature by simply hovering over any of the links. You definitely made my day today with this. Thank you.
good .. teh articles were very long
It was fine, although I’m not sold on the value of the hover feature--if it’s showing me the exact titles of articles that the hyperlink goes to, in most cases those don’t seem different enough from the linked word itself to make much difference. In general though, I love wikipedia and go there at least several times a week. I even recommend it to my students.
My experience was a favorable one. I have used Wikipedia often, however I haven’t used it in some time. I felt refreshed in my attitude, towards using Wikipedia. I forgot about all the detailed information and the great reference tools I have available to me at no cost.
IT was a great experience i had no idea that more information was available.

My experience was easy and fun. I also learned about some features I did not know about!
It was interesting to see the new little ’summary-like’ pop up that they are incorporating. I like that they are including pictures with the text of the pop-ups.
Interesting! I learned a lot about cacti and ghosts that I was not previously aware of. Everything was categorized in easy to understand sections, which made the pages easy to navigate. I have never played around with the beta functions before, and after I enabled them, I really didn’t notice much of a difference on the page. It is possible I never took an action that would display a beta feature, however. Overall, my experience was like most times I visit Wikipedia - extremely helpful and satisfying.
Great experience with little known beta features. Really liked the page preview setting!
My experience was good. Wikipedia has always been easy to navigate. The sections of each article make it easy to find the info what you need.
Great, I enjoyed the features that I tried.
interesting. I learned more than I imagined.

2. What would you call the thing that appears when you place your cursor over a link?

Answer
I would call it an "information box" or "information card".
A preview of the link.
A hovercard.
I like "preview pane", even though I can use just to provide me with some quick information, in case I am not quite familiar with a term used in the article I am reading.
I refer to it as a popup
I might call it a ’hover hint’; in this case one telling you the title of the linked article.
I will call them hyper links
a prompt information window

I would call it an "information box" or "information card".
I would call it a ’summary pop-up’ myself.
I would call it a tooltip.
Page preview
I would call this a page preview.
bubble
index box or information box

3. Do you have any feedback about what you saw appear when you held your cursor over a link?

Answer
I absolutely loved that feature. I did not know you could enable this, and I think it’s great for someone who is trying to learn about something. A lot of times there are words or nouns in articles that we aren’t familiar with, so this is a great way to improve the process of learning on Wikipedia.
I like the feature. It gives a good summary of the link.
It was generally very helpful but seemed redundant on certain pages when the item was already explained.
It was great. Best change I have seen in the years I have been using Wikipedia. I liked that the popup is stable enough to make it easy for me to right-click on the preview, in case I wish to open up that page.
it seems to be very useful You can get a preview of what is there on that page before you actually go to it
See my earlier answer. It seems harmless but the value is questionable, since most of the article titles are close enough to the link words if not identical.
Yes, I was able to get. further information on the words that I hover over by clicking the link.
it was really useful and very discriptive.

I absolutely loved that feature. I did not know you could enable this, and I think it’s great for someone who is trying to learn about something. A lot of times there are words or nouns in articles that we aren’t familiar with, so this is a great way to improve the process of learning on Wikipedia.
I like the idea of the pop-up but it needs to be a more concise pop-up that truly summarizes whatever the link is about. And the site needs to decide if it will be ’in general’ or specific to the particular ’place’/article it’s being used in.
This has the potential to be extremely helpful, and may decide whether or not I ultimately click the link. However, the example tooltip that was given in this survey was not very helpful, as the text it contained was also in the same sentence as the link.
A stylized window showing a preview of the page that the link points to. Great feature and will keep this enabled in my account.
The picture takes up quite a bit of the preview. I would prefer more informational text rather than the picture.
I liked that a photo was included with a paragraph about the topic.
It was really helpful. I liked the pictures too.

4. Which variation of this feature do you prefer? Rank in order of most favorite to least favorite.

Round 1:

Answer	1	2	3	Mean	StDev	Score	Count
Version A	75% (6)	13% (1)	13% (1)	1.38	0.7	21	8
Version B	0% (0)	75% (6)	25% (2)	2.25	0.43	14	8
Version C	25% (2)	13% (1)	63% (5)	2.38	0.86	13	8

Round 2:

Answer	1	2	3	Mean	StDev	Score	Count
Version A	57% (4)	14% (1)	29% (2)	1.71	0.88	16	7
Version B	29% (2)	43% (3)	29% (2)	2	0.76	14	7
Version C	14% (1)	43% (3)	43% (3)	2.29	0.7	12	7

5. Why did you rank the variations from the previous question the way that you did?

Answer
I like the biggest picture, so the box with the full picture and words below was my favorite.
I like seeing an image of what the link is referring to. I like the bigger image for the preview.
I like the larger picture version. The smaller picture was my last choice because I think the picture helps to explain the item and is eye-catching. It makes me want to read more about the article or item.
I like bigger pictures and I like that the text is not disrupted by the picture in Version A. But, I like all 3 versions and would not have a problem using this feature no matter which version you decide to use.
I actually liked as much information as possible to show For me the large pop up window was useful
I like Version C best because it showed me the image, but not in a way that dominated the textual summary. Btw, the beta I evaluated never showed me any of these summary capsule descriptions as hovers--was it supposed to?
I think I ranked the way I did because of the way in which the images and text was displayed and the size of the images and fonts. Also where the image was placed.
I like the ability to see a good picture of the item im looking at.

I like the biggest picture, so the box with the full picture and words below was my favorite.
I like the one where there’s a solid picture block above the text the best because it make it easy to read quickly.
I ranked A #1 because the picture is better, allowing me to see more detail without losing any of the text information. I chose C as #2 because I like the way the text wraps around the image. I chose B last because it did not stand out as much as the other two - however, B and C are roughly equivalent, and I could see preferring B over C with a different result, depending on how the text formats around the image.
The top ranked variation shows a condensed pop-up that seems to fit better with the design and layout of the site.
I liked the versions with the smaller picture and I like it best (for aesthetic reasons) with the picture on the left.
Like the larger photo
easier for my eye to read

6. Do you have any other feedback?

Answer
It was overall, a fun and easy experience.
I think this preview feature should be available for anyone!
The help section for the hovercards could be a bit more robust.
Again, thank you for a great time. Now I am going to check out the other beta features that I checked off to see what else I can do with them.
No
If the hover ’hint’ really is supposed to show you that summary, as in the 3 variants, then it could be considerably more useful than just the one-word hints I was getting on the actual beta site.
N/A
none

It was overall, a fun and easy experience.
It’s a good idea to use this ’summary pop-up’ but it has to be well written/concise.
I wish I did, but everything seems to be exactly as I expect. I could not give any more recommendations, as the features available already exceed my expectations.
Beta features should be exposed to more users by using a larger link or banner to promote. Something like "try new BETA features today", etc.
It would be nice if suggested articles were more prominently displayed, so I could have more of a StumpleUpon experience.
none
No. It was helpful to HAVE THE boxes