Extension:GuidedTour/User testing

Test scenario one
Note: this was a test of the 'gettingstarted' tour built to support version one of onboarding new Wikipedians, where we asked English Wikipedians to try their hand at copyediting an article tagged for that issue.

Test script:

"This is a test of a new interface to help people make their first edits to Wikipedia. Imagine you've decided that, in addition to reading Wikipedia, you want to help build it. You'll sign up for a free account, and then we'll send you to a place to get started. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible; tell us your thought process during each task, and try to explain your choices."


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer.)
 * 2) Go to {random copyediting article with ?tour=gettingstarted}. This is a real Wikipedia article, one of a random selection for new users who are ready to try their hand at editing, but who need an idea of something to improve. We'll provide you with a basic tour of how to do so.
 * 3) Follow the instructions provided to improve this page. If you get stuck, don't worry. Just explain what is confusing or where you get stuck, and feel free to move on to the next step.
 * 4) When you've either made an edit (or decided that you can't figure out how), think about where would you look to find more to do on Wikipedia. Explain what you would do and why.

Test A



 * Did you edit Wikipedia before this test, even once?: Nope, I have never edited or signed up to Wiki before.


 * What frustrated you the most? What improvements would have made the process easier?: I am sorry, but pretty much everything! I really like using Wiki and was surprised at how confusing the process I was take through made it seem to edit and contribute! The sign up process was straight forward, but after I had done this, I didn't see the purpose of the page I was taken through. You need firstly to be taken to the Training section, and given a walk through step by step of what it means to contribute and edit. Then you need an interactive page that talks you through step by step what every part of the page is. You need to explain what html is, how to edit it, what it looks like in 'real preview' what process there is for checking. And after the tutorial you need to show how to find pages that need editing, or is it just that I can edit any page I like??


 * What did you like about the process, if anything?: Personally, as stated on q2, I love wiki and find it an interesting and informative site. If I am helping my son with his homework, we always come to Wiki to look for information. I therefore love the site, but I hated how I was then 'left in the dark' almost to find out how to edit a page! Sorry

Test B



 * Did you edit Wikipedia before this test, even once?: No


 * What frustrated you the most? What improvements would have made the process easier?: That the tool that was supposed to teach me how to edit got stuck following the first stage. From there, I didn't know how to proceed without causing more damage than good.


 * What did you like about the process, if anything?: It seems to have great potential, compared to the old one (which is a huge article with a list of other things to read), if it had only worked.

Conclusions
Issues identified in the tests:


 * Unlike users who opted to click "Getting Started" tasks, these testers had low motivation and context, which no doubt increased confusion. Tester A in particular didn't understand why she was being directed to this particular article, which is a problem obviously solved in production considering that users opt in to choosing an article from the task list.
 * Tester B failed to see the Preview step because it was below the fold and assumed the tour was over.
 * Tester A to see first step due to outside click
 * Neither tester was sufficiently prepared for the shock of wikitext, and even when they figured out how to replicate it (tester B in particular) they were clearly impeded by it.
 * The final step was clearly insufficient to suggest to either tester that there was value in returning to Getting Started. Testers actively looked for tasks but either didn't see or didn't understand that call to action, based on the copy provided.

User experience enhancements:


 * Consider an A/B test with VisualEditor, even in its current state, to measure conversion rates and how it impacts user dissatisfaction.
 * Design a guider after step two (click edit) and before the third step (preview) explaining to the user that they've entered edit mode and will see wikitext. Also consider opening the Help toolbar and showing it.
 * Force scroll the user to the tour element in a step if it's outside the viewport (i.e. fix the below the fold issue) (Trello).
 * Consider removing the Preview step and simply describe the optional preview in the copy of the save step.
 * Rewrite the GettingStarted call to action in the final step.
 * Disable  for the first step.