Study authors: Sherah Smith & Nirzar Pangarkar
Stakeholders: Mobile Web
Phabricator Ticket: https://phabricator.wikimedia.org/T147641
In ongoing efforts to reshape the mobile web browsing experience, some branding changes have been taking place. A series of tests are going to measure the usability of said changes.
- Gather usability feedback: We want to make sure that the Menu and Search buttons on the left side of the top bar on mobile web are easily switched between, and that the user’s goals for each action are readily and easily achieved on the first try. This will inform how we iterate the layout of the top bar on mobile web.
- Gather branding feedback: Do users recognize the brand? Do they recognize they are inside Wikipedia?
- Another side-goal is to test out the execution of smaller usability tests, in order to streamline the process and make it easier for teams to rely on simple, quick feedback from the UserZoom platform.
- Do users recognize that they are inside Wikipedia? (Assess brand awareness level)
- Can a user navigate to the side menu on first attempt?
- Can a user navigate to a search on first attempt?
- Which is more common: users try to tap the magnifying glass to initiate a search, or users tap inside the search box to initiate a search?
UserZoom remote unmoderated usability test. 8 participants.
Readers, half men, half women. Minimum 6 participants.
- Gather info/assets: 10/17-10/19
- Build test inside UZ & get designer signoff: 10/20
- Recruiting: 10/20-10/21
- Test results come in/QA’d: 10/24-10/26
- Analysis/Synthesis: 10/27-10/28
- Delivery: 10/31 or first available date w/stakeholders
- (Starts with this page: https://trending.wmflabs.org/en.wikipedia/Halloween) Tell me, out loud, a little about what you’re seeing here on this page. Take a minute to poke around a little.
- Thanks for your descriptions. Next, please search for the word “Sloth”. Say aloud what you are doing, and describe what happens when you have performed the search for “Sloth.”
- Now search for “Octopus.” Say aloud what is happening, and describe what you see after you have performed the search.
- Now go to the main menu. What are the options listed in the main menu, and (without selecting the options) what do you think they would lead you to?
- Now close the menu. Click on any interesting link in the page you see when you close the menu. Describe where you are on the internet, and what the pages you’ve been viewing are for.
- Open the main menu once again, and then close it. What do you think about the menu?
Follow up Questions
- Did you think it was easy to search for something?
- What did you think you were testing today?
- Do you have any feedback about what you saw today?
Answers to Research Questions
Do users recognize that they are inside Wikipedia? (Assess brand awareness level)
Based on the feedback from the video recordings as well as survey responses, we feel confident that participants do understand that they are using Wikipedia, rather than browsing "the Internet" or "a search engine."
Yes. There were no issues locating the menu. One out of 16 people had issues understanding how to close the menu, so we are rating its usability as an A+.
Yes. There were zero issues locating and using the search bar throughout the tests.
Which is more common: users try to tap the magnifying glass to initiate a search, or users tap inside the search box to initiate a search?
Only one person tapped the magnifying glass inside the search bar on Prototype A. There were no usability issues with tapping into and using the search bar throughout the tests.
Final Questionnaire Answers
There are two batches of answers. The first tables come from Prototype version A tests, the second from Prototype version B.
1. Did you think it was easy to search for something?
|Yes. The search was at the top of every wiki page, so it’s easy to search for what your looking for, because you never have to exit the page you’re on. Also, whole you’re typing, wiki looks through its database to find what you’re possibly searching for. This is especially good if you’re not sure how to spell something, but you have the 1st few letters.|
|Yes it is as easy as any other search on the internet.|
|Yes, and I loved the suggestive searching.|
|It was easy to search but it took a long time to load.|
|It’s super easy to search for things. No hassle whatsoever|
|Yes it was easy to search|
|Yes it was very easy to search for articles on topics I was interested in.|
2. What did you think you were testing today?
|The look and accessibility of mobile Wikipedia.|
|The Wiki mobile website, and the ease of navigating and searching using it through Wikipedia and its database.|
|I think I was testing the ease of a Wikipedia search and the main menu.|
|User experience of the web app|
|Wikipedias new memu.|
|Wikipedia’s search engine|
|The Wikipedia site|
|The Wikipedia mobile page|
|I’m not sure. Just a new Wikipedia home page|
|The search engine for Wikipedia|
|The mobile page for Wikipedia. I believe I was testing usability, functionality, layout, navigation|
|Wikipedia search function|
|Functionality of the Wikipedia site|
|The main search and navigation settings within the mobile site for Wikipedia.|
|New menu and features such as random and article of the day|
3. Do you have any feedback about what you saw today?
|Nope. I think the mobile page is great!|
|I was confused a little bit on question 5 and 6 of the test referring to main menus, links and where I was on the Internet but maybe I was doing right after all. I was not %100 sure though.|
|Wikipedia web app is mobile friendly, it could be better in terms of responsiveness. Loading pages happened slowly. There was a small bug when I typed in a query and it didn’t change the UI.|
|I loved the new look Wikipedia. Clean design, and I love how they are trying to give Wiki more of a reddit feel.|
|Make directions more specific|
|It was a bit slow|
|The main menu was vague.|
|I didn’t clearly see an option to complete the last task of going to the homepage and closing it. I was a bit confused.|
|I really enjoyed today’s study,|
|I loved the options that pop up as you’re searching|
|Fairly easy to navigate and find articles on this website.|
|Everything worked very easy, and the navigation was simple, clean, and effective.|