Extension:GuidedTour/User testing

Test scenario one
Note: this was a test of the 'gettingstarted' tour built to support version one of onboarding new Wikipedians, where we asked English Wikipedians to try their hand at copyediting an article tagged for that issue.

Test script:

"This is a test of a new interface to help people make their first edits to Wikipedia. Imagine you've decided that, in addition to reading Wikipedia, you want to help build it. You'll sign up for a free account, and then we'll send you to a place to get started. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible; tell us your thought process during each task, and try to explain your choices."


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer.)
 * 2) Go to {random copyediting article with ?tour=gettingstarted}. This is a real Wikipedia article, one of a random selection for new users who are ready to try their hand at editing, but who need an idea of something to improve. We'll provide you with a basic tour of how to do so.
 * 3) Follow the instructions provided to improve this page. If you get stuck, don't worry. Just explain what is confusing or where you get stuck, and feel free to move on to the next step.
 * 4) When you've either made an edit (or decided that you can't figure out how), think about where would you look to find more to do on Wikipedia. Explain what you would do and why.

Test A



 * Did you edit Wikipedia before this test, even once?: Nope, I have never edited or signed up to Wiki before.


 * What frustrated you the most? What improvements would have made the process easier?: I am sorry, but pretty much everything! I really like using Wiki and was surprised at how confusing the process I was take through made it seem to edit and contribute! The sign up process was straight forward, but after I had done this, I didn't see the purpose of the page I was taken through. You need firstly to be taken to the Training section, and given a walk through step by step of what it means to contribute and edit. Then you need an interactive page that talks you through step by step what every part of the page is. You need to explain what html is, how to edit it, what it looks like in 'real preview' what process there is for checking. And after the tutorial you need to show how to find pages that need editing, or is it just that I can edit any page I like??


 * What did you like about the process, if anything?: Personally, as stated on q2, I love wiki and find it an interesting and informative site. If I am helping my son with his homework, we always come to Wiki to look for information. I therefore love the site, but I hated how I was then 'left in the dark' almost to find out how to edit a page! Sorry

Test B



 * Did you edit Wikipedia before this test, even once?: No


 * What frustrated you the most? What improvements would have made the process easier?: That the tool that was supposed to teach me how to edit got stuck following the first stage. From there, I didn't know how to proceed without causing more damage than good.


 * What did you like about the process, if anything?: It seems to have great potential, compared to the old one (which is a huge article with a list of other things to read), if it had only worked.

Conclusions
Issues identified in the tests:


 * Unlike users who opted to click "Getting Started" tasks, these testers had low motivation and context, which no doubt increased confusion. Tester A in particular didn't understand why she was being directed to this particular article, which is a problem obviously solved in production considering that users opt in to choosing an article from the task list.
 * Tester B failed to see the Preview step because it was below the fold and assumed the tour was over.
 * Tester A failed to see first step due to outside click
 * Neither tester was sufficiently prepared for the shock of wikitext, and even when they figured out how to replicate it (tester B in particular) they were clearly impeded by it.
 * The final step was clearly insufficient to suggest to either tester that there was value in returning to Getting Started. Testers actively looked for tasks but either didn't see or didn't understand that call to action, based on the copy provided.

User experience enhancements:


 * Consider an A/B test with VisualEditor, even in its current state, to measure conversion rates and how it impacts user dissatisfaction.
 * Design a guider after step two (click edit) and before the third step (preview) explaining to the user that they've entered edit mode and will see wikitext. Also consider opening the Help toolbar and showing it.
 * Force scroll the user to the tour element in a step if it's outside the viewport (i.e. fix the below the fold issue) (Trello).
 * Consider removing the Preview step and simply describe the optional preview in the copy of the save step.
 * Rewrite the GettingStarted call to action in the final step.
 * Disable  for the first step.

Test scenario two
In this test, we wanted to know what the impact would be if we changed from text buttons ("Next", "Okay") to the use of symbols (forward arrows, check marks). Will they make next steps less clear for users?

Test script
This is a test of a new help system for Wikipedia. In this scenario, imagine that you've decided to sign up for an account on the site in order to help expand and revise the content. This task is on a test version of Wikipedia, so don't be concerned about making correct changes to pages. Remember, it's the software we're testing, not you!


 * Tasks
 * 1) Log in to the test site using the following credentials:
 * 2) Once you're logged in visit [en.wikipedia.beta.wmflabs.org/wiki/Corgi?tour=firstedit this test page].
 * 3) You should be on a test version of a Wikipedia article. What do you think you're being asked to do here? Briefly describe what you would do next and why.
 * 4) If you've guessed that you're being asked to make an edit to this page, you're correct! Try to complete the task by editing the Corgi page. If you can't, don't worry, this is optional.


 * Questions
 * 1) What frustrated you most? What would you change if you could?
 * 2) What did you find most helpful about this experience?
 * 3) May we share your video publicly with other Wikipedia users? This will help us show how to improve Wikipedia's editing process for new contributors.

Highlights
In the older, text-based version, users seemed to have no trouble navigating and understanding the tour for the most part.

Conclusions

 * There is no reason for the Preview step to not proceed directly to the Save step.
 * Showing the Preview and Save guiders before the user had the chance to complete the prerequisite step (entering text or viewing the preview) is potentially confusing or annoying for users. We should explore delaying both items, either by adding an intermediate step or waiting for the user to enter wikitext.
 * Several users missed the fact that there was a step after being asked to edit the entire page. This isn't bad, but we do want to educate the user about the option to edit just a section. We might consider combining the "edit page" and "edit section" steps in to one (such as by highlighting both buttons at once), or rewriting the copy to emphasize that editing the entire page is optional.