Extension:GuidedTour/User testing

Test scenario one
Note: this was a test of the 'gettingstarted' tour built to support version one of onboarding new Wikipedians, where we asked English Wikipedians to try their hand at copyediting an article tagged for that issue.

Test script:

"This is a test of a new interface to help people make their first edits to Wikipedia. Imagine you've decided that, in addition to reading Wikipedia, you want to help build it. You'll sign up for a free account, and then we'll send you to a place to get started. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible; tell us your thought process during each task, and try to explain your choices."


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer.)
 * 2) Go to {random copyediting article with ?tour=gettingstarted}. This is a real Wikipedia article, one of a random selection for new users who are ready to try their hand at editing, but who need an idea of something to improve. We'll provide you with a basic tour of how to do so.
 * 3) Follow the instructions provided to improve this page. If you get stuck, don't worry. Just explain what is confusing or where you get stuck, and feel free to move on to the next step.
 * 4) When you've either made an edit (or decided that you can't figure out how), think about where would you look to find more to do on Wikipedia. Explain what you would do and why.

Test A



 * Did you edit Wikipedia before this test, even once?: Nope, I have never edited or signed up to Wiki before.


 * What frustrated you the most? What improvements would have made the process easier?: I am sorry, but pretty much everything! I really like using Wiki and was surprised at how confusing the process I was take through made it seem to edit and contribute! The sign up process was straight forward, but after I had done this, I didn't see the purpose of the page I was taken through. You need firstly to be taken to the Training section, and given a walk through step by step of what it means to contribute and edit. Then you need an interactive page that talks you through step by step what every part of the page is. You need to explain what html is, how to edit it, what it looks like in 'real preview' what process there is for checking. And after the tutorial you need to show how to find pages that need editing, or is it just that I can edit any page I like??


 * What did you like about the process, if anything?: Personally, as stated on q2, I love wiki and find it an interesting and informative site. If I am helping my son with his homework, we always come to Wiki to look for information. I therefore love the site, but I hated how I was then 'left in the dark' almost to find out how to edit a page! Sorry

Test B



 * Did you edit Wikipedia before this test, even once?: No


 * What frustrated you the most? What improvements would have made the process easier?: That the tool that was supposed to teach me how to edit got stuck following the first stage. From there, I didn't know how to proceed without causing more damage than good.


 * What did you like about the process, if anything?: It seems to have great potential, compared to the old one (which is a huge article with a list of other things to read), if it had only worked.

Conclusions
Issues identified in the tests:


 * Unlike users who opted to click "Getting Started" tasks, these testers had low motivation and context, which no doubt increased confusion. Tester A in particular didn't understand why she was being directed to this particular article, which is a problem obviously solved in production considering that users opt in to choosing an article from the task list.
 * Tester B failed to see the Preview step because it was below the fold and assumed the tour was over.
 * Tester A failed to see first step due to outside click
 * Neither tester was sufficiently prepared for the shock of wikitext, and even when they figured out how to replicate it (tester B in particular) they were clearly impeded by it.
 * The final step was clearly insufficient to suggest to either tester that there was value in returning to Getting Started. Testers actively looked for tasks but either didn't see or didn't understand that call to action, based on the copy provided.

User experience enhancements:


 * Consider an A/B test with VisualEditor, even in its current state, to measure conversion rates and how it impacts user dissatisfaction.
 * Design a guider after step two (click edit) and before the third step (preview) explaining to the user that they've entered edit mode and will see wikitext. Also consider opening the Help toolbar and showing it.
 * Force scroll the user to the tour element in a step if it's outside the viewport (i.e. fix the below the fold issue) (Trello).
 * Consider removing the Preview step and simply describe the optional preview in the copy of the save step.
 * Rewrite the GettingStarted call to action in the final step.
 * Disable  for the first step.

Test scenario two
In this test, we wanted to know what the impact would be if we changed from text buttons ("Next", "Okay") to the use of symbols (forward arrows, check marks). Will they make next steps less clear for users?

Test script
This is a test of a new help system for Wikipedia. In this scenario, imagine that you've decided to sign up for an account on the site in order to help expand and revise the content. This task is on a test version of Wikipedia, so don't be concerned about making correct changes to pages. Remember, it's the software we're testing, not you!


 * Tasks
 * 1) Log in to the test site using the following credentials:
 * 2) Once you're logged in visit [en.wikipedia.beta.wmflabs.org/wiki/Corgi?tour=firstedit this test page].
 * 3) You should be on a test version of a Wikipedia article. What do you think you're being asked to do here? Briefly describe what you would do next and why.
 * 4) If you've guessed that you're being asked to make an edit to this page, you're correct! Try to complete the task by editing the Corgi page. If you can't, don't worry, this is optional.


 * Questions
 * 1) What frustrated you most? What would you change if you could?
 * 2) What did you find most helpful about this experience?
 * 3) May we share your video publicly with other Wikipedia users? This will help us show how to improve Wikipedia's editing process for new contributors.

Highlights
In the older, text-based version, users seemed to have no trouble navigating and understanding the tour for the most part.

Conclusions

 * There is no reason for the Preview step to not proceed directly to the Save step.
 * Showing the Preview and Save guiders before the user had the chance to complete the prerequisite step (entering text or viewing the preview) is potentially confusing or annoying for users. We should explore delaying both items, either by adding an intermediate step or waiting for the user to enter wikitext.
 * Several users missed the fact that there was a step after being asked to edit the entire page. This isn't bad, but we do want to educate the user about the option to edit just a section. We might consider combining the "edit page" and "edit section" steps in to one (such as by highlighting both buttons at once), or rewriting the copy to emphasize that editing the entire page is optional.

= Guerrilla Testing for Guided Tours = The goal of this research was to observe people interact with the Guided Tours beta page and determine:
 * 1) whether users realized that the page was a 'tutorial'
 * 2) how users interact with the tour page
 * 3) whether users understood the different ways to edit (full page, section-specific), how to edit, how to preview their edit, how to confirm/save their edit

To get started, we asked people a few basic questions first:
 * 1) What kind of phone do you use/what OS?
 * 2) Do you use a tablet, and if so, what model/OS?
 * 3) Do you use a computer, and if so, what model/OS?
 * 4) Do you ever use Wikipedia? (Goal of this question is to find out if they know you can edit wikipedia or not. If they answer, "Yes, I use Wikipedia", we ask "how do you use it?" If they describe that they read, mostly and don't mention editing, then we ask "Do you know that you can edit Wikipedia?"

We presented users with the guided tours beta page and allowed them to interact with the tour pop-ups. Ultimately, we directed them to make an edit (add a sentence in a particular section, preview it, and save/confirm that edit).

Where
Thursday, June 26, 2014: We went to Specialty's so we could use wifi there.

June 26, 2014

 * 1) Users tend to view the hover as a call to action, not as guidance. Even when asked specifically to read and then explain what they believe to be the purpose of the hovers, they give responses that seem to suggest that the hovers are prompting them to edit.

June 26, 2014

 * 1) The guided tour has a bit of an identity crisis. It's not a pure module, and it doesn't really guide unless the user somehow follows the invisible set of steps the creator has in mind for him/her. Without true guidance, the user inevitably will veer off course, and often does as shown in the guerrilla session.
 * 2) Users often are motivated by the first hover to just click edit, and hence misses out the secondary tour hover.
 * 3) Preview changes hover appears sometimes after users save edit and return to the main corgi page.
 * 4) After clicking preview, the next page seems inconsistent after testing. Sometimes it brings users to the top of the page (where the preview can be seen), or it jumps down to where the edit box is so that the user can see the confirm/save edit hover prompt.
 * 5) Users cannot click show preview button when the hover is still active, same for save button.
 * 6) Incorporate see diff into preview somehow.
 * 7) Show preview button puts user in the part of the page where the preview is viewable, instead of jumping user down to the edit box part of the page.
 * 8) Make show preview and save buttons clickable even when the hovers are up.
 * 9) Perhaps the guided tour should be more obvious - grayed out page with just the hover buttons looking 'active. Nothing else is clearly legible or clickable until after users read and click through the two hover menus. After clicking through the two, users can click on 'edit'. Then users can explore the edit page - perhaps everything can be grayed out after something is typed in the edit window for preview hover to come up. No need to gray anything out for the show preview page, keep save hover as is.

Man (26-35)
Task
 * Android phone, iOS/Android tablets, Windows/Linux/Mac computers
 * Reads Wikipedia, knows about editing but doesn't
 * regarding the top hover, he comments that wiki can allow comprehensive, but not point edits. clicks next button
 * realizes that there are section-specific means of editing as well. clicks edit next to 'Appearance' section
 * types sentence where directed
 * would choose to click preview only if he'd made a 'complicated' or 'bulk' change. clearly wants to just go ahead and click save. clicks preview button upon tester prompting
 * he remarks that preview just brings him back to the edit window where he made the edit (this is a buggy thing. that was just a chance occurrence, and the preview brings user back to the edit window, not the actual preview part of the page)
 * comments that he can't really see his changes easily in preview - he wants something more distinct, wants the changes highlighted or similar
 * wants changes 'called out', essentially wants 'see diff' in preview mode

Woman (26-35, 36-45)
Task
 * iPhone, iPad, Mac computer
 * Reads Wikipedia, knows about editing but doesn't. Wants to (esp. Arabic wikipedia), but has blockers
 * reads the first hover and excitedly clicks edit button right away
 * scrolls around quickly and immediately begins typing away at one of the bulleted lists on the page, getting right to it!
 * clicks out of the explore preview hover
 * asks and reads about the edit summary, talks about how it is for the user to record notes and explain changes
 * clicks the minor edit box, then clicks save. very exploratory, but perhaps also a very impulsive user in a way
 * bug - preview changes hover menu appears on the corgi page after this user saved her edit. confusing
 * this user provided her email address for further contact on similar testing

Man (15-25)
Task
 * iPhone, iPad, doesn't use computers much
 * Reads Wikipedia, knows about editing but doesn't "haven't thought about it"
 * immediately starts exploring the page as he usually would, clicking on a section in the ToC
 * after tester prompting, user reviews the top hover and clicks next to see the second hover as well
 * goes to edit menu and makes the prompted change
 * reviews the preview hover, and clicks preview
 * appears to jump up the page after clicking preview - inconsistent, or he scrolled up quickly himself, as typically it brings the user down to the edit box so user can see the save hover prompt
 * user attempted to click show preview button a couple times when the hover was up, and couldn't. had to click out of hover prompt first
 * ultimately saved page and confirmed the edit

Man (15-25)
Task
 * iPhone, iPad, Macbook
 * Reads Wikipedia, knows about editing. Edits, but a long time ago.
 * went right to edit
 * interpreting first hover as a call to action, not an 'fyi' thing or a 'guide', necessarily
 * user experiences inability to click on action buttons when hover dialogue is up (similar to above)
 * after user saves edit, the preview hover pops up on corgi page (similar to above)