Reading/Web/Accessibility for reading/Repository/Usability testing

This page presents a usability testing. The goal was to determine discoverability and usability of text, page width, and color accessibility options on desktop web and mobile web. For desktop, the text and page width options were tested on the Vector 2022 skin. For mobile, the text and color options were tested on the Minerva skin. The testing was done by Daisy Chen.

Research questions and Recommendations[edit]

Questions[edit]

Discoverability: Do participants notice accessibility settings, and/or know where to look for them?
Icon, labels: Does the appearance menu icon resonate with participants? Do the various option labels make sense for participants?
Intuitiveness and usability: Is the interaction with the appearance and settings menus smooth/intuitive?

Vector 2022[edit]

Element	Recommendation
Location/discoverability	Open on sidebar by default, and it can be hidden/pinned (and first-timers will get the same tooltip pop-up, but making sure it is visible to them even if they are scrolled lower on the page).
Icon	Could consider a slightly larger icon if it fits with UI standards. Could consider additionally an icon change; though this is not a huge concern especially if the sidebar is open by default first, users did not generally correlate it with accessibility settings.
Labels	Though participants didn’t indicate any specific concerns or issues, other tests have suggested that a change from ‘Text’ to ‘Text size’ would be more clear and specific.
Dynamic display	Currently, one cannot pin the menu to the sidebar if the browser width is too narrow, and the width option is not shown unless the browser exceeds a certain width. Consider the experience of showing options (grayed out when not applicable) regardless, for the sake of awareness.

Minerva[edit]

Element	Recommendation
Text size	Consider displaying the different text sizes with slightly more contrast among the options, as some users felt the difference to be negligible.
Labels	Consider changing ‘text’ to ‘text size’, and changing ‘color’ to the ‘day/night mode’ or ‘screen color’ to better match user expectations. Also, consider adding some explanation as to what ‘automatic’ means for better user understanding.
Location	Some users were slightly confused that these were the entirety of the ‘settings’ section; if/when there are enough/too many settings, add a subsection for these accessibility options.
Other suggestions	Consider adding some font options; the participant with ADHD made a good case for doing so! Researcher note: additionally, though the test didn’t necessitate reviewing an article, I would still suggest, if technically possible, a smoother and less clumsy transition between the menu screens, selecting accessibility options, and returning to the article page.

Methodology[edit]

Userlytics.com (unmoderated remote testing with participants from vendor's participant panel)
Test environments
- Desktop web (Vector 2022) test link
- Mobile web (Minerva) test link

Participants[edit]

Screener elements

Wikipedia user
reader/non-editor
not a developer/engineer/user researcher
(optional) user of accessibility feature(s)
Vector 2022: laptop/desktop (with an 15” diagonal or larger screen) user
Minerva: mobile/tablet user

Demographic breakdown

Vector 2022: 2 Brazil, 3 Australia; 5 men
Minerva: 6 UK, 5 women, 1 man

Protocol screener[edit]

(Bold indicates go-ahead)

For both skins: Which of the following sites have you used to find and/or access information online? Please select all that apply.
- Twitter
- Reddit
- Wikipedia
- Quora
- Facebook
For both skins: When you access Wikipedia, what kind of device do you use primarily?
- For Vector 2022:
  - Only on computer (laptop or desktop)
  - Only on mobile device (phone or tablet)
  - A mix of both (desktop + mobile)
- For Minerva:
  - Only on computer (laptop or desktop)
  - Only on mobile device (phone or tablet)
  - A mix of both (desktop + mobile)
What device will you be using today to complete this test?
- For Vector 2022:
  - Laptop
  - Desktop
  - Tablet
  - Mobile device
  - Other
- For Minerva:
  - Laptop
  - Desktop
  - Tablet
  - Mobile device
  - Other
For Vector 2022: What size screen are you using with this device?
- 13” diagonal or smaller
- 15” diagonal
- 17” diagonal or larger
For both skins: What do you typically do on Wikipedia? Please select all that apply.
- I moderate content on Wikipedia.
- I contribute content to Wikipedia extensively.
- I contribute content to Wikipedia sometimes.
- I haven’t edited or contributed content to Wikipedia.
- I don't know // I didn’t know that Wikipedia articles can be edited.
- I read Wikipedia articles.
For both skins: In what field(s) have you worked or received a degree? Please select all that apply.
- Software engineering
- Finance and administration
- Human Resources
- User experience research and design
- Corporate law
- None of these
For both skins: Have you ever used accessibility settings (changing text size and other display preferences) on your device(s)?
- Yes (ideal. Other answers also ok)
- No
- I’m not familiar with accessibility settings.

Protocol tasks[edit]

Vector 2022[edit]

Today you’ll be helping us test out a new feature on Wikipedia. Please remember to speak aloud any reactions, thoughts, and/or feedback as you move through the tasks!
Please navigate to the Dog article using this link and take a quick look around on the page. Narrate what you’re seeing on the page.
Have you noticed any page elements you haven’t seen on a Wikipedia page before? No worries if not.
If you don’t see anything new, scroll to the top of the page and look to the icon next to your account name (if you’re logged in) or the ‘create account’ link (if you’re not logged in). What does this icon represent to you? What do you think of this icon?
Now, click on the icon. Next, take a look at where it says ‘move to sidebar’ on this menu. What do you think this does? Try it out (or if you already have), let us know what you think.
You should see the ‘Appearance’ column on the right side of the page. Now, let’s take a look at some of the options in the column. What do you think of these? What did you expect to happen when you make different selections on the ‘Text’ option here? If you’ve already checked these options out, how do the different size options compare to your expectations? What do you think of the ‘Text’ label?
Do you ever change text size when you’re reading/browsing the internet on your laptop/desktop computer, and have you ever explored or changed your device text size settings? If yes for either and if you feel comfortable sharing, why did you change it, and how was the experience of doing so?
Now, explore the ‘Width’ options (they should be located right below the text size options; if you can’t see anything, make your browser window as wide/large as you can, and the width options should then appear). What do you think of the options, and how they are rendering on your browser/screen?
Did you have to enlarge your browser to see the width options? And generally, what do you think of the width options you reviewed? How do you feel about the ‘Width’ label for these options?
Have you ever changed page width when you’re reading/browsing the internet on your laptop/desktop computer?
How might you remove this Appearance menu from view if you’re done with your selections or don’t need to use it anymore?
Next (if you haven’t already), look to where it says ‘hide’ on the Appearance menu. What do you think that does? Try clicking on it and let us know what you think.
What happened after you clicked on ‘hide’? Is that what you expected to happen? What would you prefer (if different)?
If you didn’t see it, scroll back to the top of the page, and the icon should have reappeared back at the top where it was originally. What do you think of ‘pinned’ version of the menu?
Generally, where would you prefer to have this menu be located as you browse Wikipedia? Currently, as you saw, the default is that the appearance menu is located near the user/account name or the ‘create account’ link. Where would you like it to be by default?
Thank you for your feedback thus far! Those were all the tasks for today, but before you go, we have just a few more final questions:
1. Do you have any thoughts you’d like to share about what you explored with us today?
2. If you have used accessibility (changing font, changing text sizes, zooming in/out, night/dark mode, etc.) settings in the past, can you tell us briefly about what settings you used/changed and (if you’re comfortable) for what purpose?
3. Is there anything you’d like for us to change or include in our appearance menu?

Minerva[edit]

Today you’ll be helping us test out a new feature on Wikipedia. Please remember to speak aloud any reactions, thoughts, and/or feedback as you move through the tasks!
Please navigate to this page using your mobile browser and take a quick look around on the page (please note that this a test wikipedia, so you will not see standard Wikipedia pages).
If you wanted to check accessibility settings (changing text size and other display preferences) on this page, how would you go about it? Show us where you would look and please narrate your thoughts.
If you haven’t already, click on the top left icon, and then go to ‘Settings’. What do you think of the options in the Settings menu; what do you expect would happen if you changed the ‘Text’ settings or the ‘Color’ settings?
Go ahead and test out the ‘Text’ options. What do you think of the experience, and does it differ from what you expected?
Have you ever changed text size when you’re reading/browsing the internet on your mobile device or tablet? Have you ever explored or changed your device text size settings? What'd you think of those past experiences?
Now take a look at the ‘Color’ options. What do you think of the experience, and does it differ from what you expected?
Have you ever changed the display settings (and which ones?) when you’re reading/browsing the internet on your mobile device or tablet? Have you ever explored or changed your device display settings (native to the device, like iOS’s night mode, or with an add-on app like Flux)?
How do you feel about these two settings being labeled ‘Text’ and ‘Color’?
Where would you prefer to have these settings be located as you browse Wikipedia on your mobile browser? Currently, as you saw, they’re located in Settings within the main menu.
Thank you for your feedback thus far! Those were all the tasks for today, but before you go, we have just a few more final questions:
1. Do you have any thoughts you’d like to share about what you explored today with us?
2. If you have used accessibility (changing font, changing text sizes, zooming in/out, night/dark mode, etc.) settings in the past, can you tell us briefly about what settings you used/changed and (if you’re comfortable) for what purpose?
3. Is there anything you’d like for us to change or include in the settings menu?

Findings[edit]

Vector 2022[edit]

Task/Element	Score	Finding
Notice anything new?	X	0 of 5 participants noticed the appearance icon. 2 mentioned the language options and 1 the table of contents.
Icon	X	1 of 5 participants correctly guessed that it represents “enhanc[ing] font or accessibility”. Other guesses included “edit”, “lock”, or “read incognito”.
Sidebar	✓	3 of 5 participants reacted positively to the sidebar option. "interesting, keep it in a fixed position" “I like it”
Text, and the Text label	✓	5 of 5 participants felt the text option met their expectations, and 2 specifically call out that the label “makes sense”.
Changing text size in general	-	4 of 5 participants say they have changed text size, with 3 of 4 using the zoom function on their browser. 1 has a special setting to do so on their mouse scroller mechanism.
Width, and the Width label	✓	4 of 5 were able to experience the width change. Only 1 experienced a significant width change. 3 participants had a positive reaction (“it fills out the screen a lot more. It's quite convenient”, “more appealing to the eyes”), but 1 felt it was “not really something that I would use. I think it's nice to have but doesn't make a difference”. 0 of 5 participants commented on the label; however, based on their interactions and commentary, it was likely quite straightforward for them.
Changing width in general	-	0 of 5 participants have adjusted for width previously.
Removing the sidebar	✓	5 of 5 participants successfully used the hide option to repin the appearance menu to the top/icon, and 4 felt it met their expectations. “Intuitive” However, 1 participant indicated he was “worried … I won’t be able to know where it is”. Researcher note: in the heuristic evaluation, I noted that if a user were to scroll further down the page and then hide the sidebar, users may experience this issue; a first-time indicator tooltip that jumps users up to the top of the page may help.
Icon/Pinned menu	-	2 of 5 participants preferred the sidebar version more due to its accessibility “easily accessible for viewing and changing options” “if someone is having hard trouble reading they can change it on the fly” 1 of 5 participants preferred the pinned version, as it was “not too distracting” and allowed for their standard reading experience. The remaining 2 participants didn’t indicate any preference, but did mention that the pinned version is very small, and was/could be missed. “I didn’t see [it] before … in the beginning” "icon could be slightly bigger, i can see myself forgetting that it's there"
Menu location	-	4 of 5 participants prefer it where it is, with 1 emphasizing that having it in the sidebar would be ideal and 1 preferring the pinned state. 1 of 5 participants wanted to see all configurations on the left panel of the page, and leaving reading and editing options on the right portion of the page. Other suggestions for location include: on the left panel, closer to the article title, near the languages dropdown, in the hamburger menu.
Additional notes/comments	-	2 participants more or less explored the entire workflow and options in the first minute or so. 3 participants were using dark mode and/or a high contrast, dark mode browser.

Minerva[edit]

Element	Score	Finding
Finding accessibility options	✓	6 of 6 participants quickly found the settings in the hamburger menu. "very intuitive and easy to understand" “easy to do”
Text, and the Text label	✓	6 of 6 participants felt the text option met their expectations to some extent, but 2 specifically call out that the differences between the text sizes were not significant enough. 5 participants suggested changing ‘text’ to ‘text size’ (4) or ‘font size’ (1).
Changing text size in general	-	5 of 6 participants say they have changed text size. Some reasons include “for mom”, “on subtitles”, and “to fit more text on screen”.
Color, and the Color label	✓	Most participants felt it met their expectations to some extent. 1 seemed unsure about what the ‘automatic’ setting was based upon. 5 participants felt ‘color’ was confusing in some way, and suggested changing to ‘day/night mode’ (2), ‘screen color’ (2), ‘dark/bright mode’ (1), and ‘background color’ (1). 1 mentioned that changing would make it “how it is in … normal phone settings”.
Changing color in general	-	5 of 5 participants (1 mistakenly skipped this task) have adjusted display settings (using dark mode, sepia filter, etc.).
Location	-	6 of 6 participants felt that these options are where they’re supposed to be located. 2 of 6 participants felt additionally, however, that the options should be within a settings subsection entitled ‘accessibility’, ‘page view’, or ‘formatting’.
Additional notes/comments	-	1 participant suggested and expounded at length about how different font options would greatly help users with ADHD consume the content better.