Onboarding new Wikipedians/user testing

We conducted three remote user tests.

Testing scenario one
We asked users to complete the following tasks.


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer. Email is optional.)
 * 2) Take a moment to scan this page [Ed. Special:GettingStarted]. What do you think this page is asking you to do? What do you expect to happen next if you click on any of the links?
 * 3) Follow the instructions provided to get started with editing. Take no more than 10 minutes. If you get stuck, don't worry. We're looking to learn how to make this experience easy for people, so detailed feedback really helps, even if you aren't sure what to do.
 * 4) If you're finished, close your browser tab or window. Imagine you might be meeting some friends for dinner, and want to end your Wikipedia session.
 * 5) Now let's say you're back from dinner and want to go back to editing Wikipedia. Open a new browser tab or window, and return to Wikipedia. Please talk through the process. If you're not logged in, please log in again.
 * 6) If you completed an edit in step #4, where would you expect to find this edit?

Test A



 * Did you edit Wikipedia before this test, even once?: No, I have never edited on Wikipedia.


 * What frustrated you the most? What improvements would have made the process easier?: The editor was very stratightforward, however it took me a few moments to see the 'edit' tab, top right. If it were easier to see I would have more quickly understood how to navigate between regular layout and edit.


 * What did you like about the process, if anything?: I liked the editor. It could have been much more imposing and intimidating, but the layout was friendly and easy to understand.

Test B



 * Did you edit Wikipedia before this test, even once?: No, never.


 * What frustrated you the most? What improvements would have made the process easier?: I didn't see anything that I would really consider instructions. There were notes but nothing to walk me through or really explain anything about the process. I would want more step by step 'how to' instructions.


 * What did you like about the process, if anything?: I like that you can edit these pages, but other than that I didn't like much because everything seemed so difficult and too big project.

Test C



 * Did you edit Wikipedia before this test, even once?: No, I did not edit Wikipedia before this test, not once.


 * What frustrated you the most? What improvements would have made the process easier?: What frustrated me the most was not being able to go back and locate my edit. I don't feel the editing process was that hard, just finding what I edited was difficult. I would improve this by adding a button to see what you've edited in the past, that would make it easier.


 * What did you like about the process, if anything?: What I like about the process was the simplicity of it. It seemed pretty easy to do the actual editing.

Conclusions

 * We specifically asked the testers what they thought Special:GettingStarted was asking them to do, and what would happen if they clicked on any of the links. All of the users understood the landing page correctly, and the request to choose a page and contribute to it.
 * While users understood the GettingStarted interface was asking them to edit, they assumed that they would have to have some prior knowledge/expertise in the subject of the articles in the list.
 * There was clearly not enough instruction about the kind of task requested and how to complete it when users were on an article they chose from the list.
 * Users had less difficulty with wikitext than one might expect, almost certainly due to the nature of the task we were asking them to consider. They generally ignored markup that was irrelevant to their work, and some even identified sections within articles. The smaller maximum length of onboarding articles, 10k bytes, may be a factor.
 * None of the users completed an edit during the test, though 2/3 of them made it to the edit screen, then described precisely what they would do. (The one user who attempted to actually save an edit ran in to the CAPTCHA delivered to users when they save an edit with an external link addition.)
 * Users all navigated to Wikipedia through the main wikipedia.org portal, using either the link to English Wikipedia or the search function to find "their version".
 * There are a number of other insights in to how users learn to navigate Wikipedia, including key functions such as search, history pages, watchlists, and more. Those processes are currently outside the scope of this project, so we won't elaborate on them yet.

Test scenario two
This was a test of the new Special:GettingStarted landing page, sans guided tours, launched on Thursday, March 7th. We asked users to complete the following tasks:

"This is a test of the signup process for Wikipedia and a new interface for getting started as an editor of the site. In this scenario, imagine you've decided to join Wikipedia to update and improve its information. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible.


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer.)
 * 2) Take a moment to scan this page. What do you think this page is asking you to do? If you want to become a Wikipedia editor, what would you do next?
 * 3) Once you've scanned the page, go ahead and try to make your first contribution as a Wikipedia editor. If you get stuck, don't worry! Just explain what you would do next and why. It's okay to say you feel like you would give up at any point."

Test A



 * Did you edit Wikipedia before this test, even once?: I've done it before, but never consciously being aware that I was being a copyeditor perse, more just clicking on edit on the top when I've noticed something needs to be hyperlinked or spell-corrected. Probably done it over a hundred times by now.


 * What frustrated you the most? What improvements would have made the process easier?: You might explain the process with some kind of information or something on that 2nd page that one comes to after signing up to sord've let them know what the next step is, that page kind've leaves it up to someone to guess what to do next.


 * What did you like about the process, if anything?: I liked editing the errors using the edit feature, works really well. The signup process was easy too.

Test B



 * Did you edit Wikipedia before this test, even once?: I have never done it before today.


 * What frustrated you the most? What improvements would have made the process easier?: Reading the strange, foreign format confused me the most. That would take some getting used to. I would suggest the editor read the article first and then go to edit. I would also show some examples of what is importatnto be edited- that would help.


 * What did you like about the process, if anything?: It is fun to improve things- I liked reading and revising.

Conclusions
Issues identified from the tests:


 * In the first two tests, we unintentionally primed users to choose the "Copyedit" task by using the terminology "Wikipedia editor". This was corrected in a third user test script.
 * There was confusion about what kind of decision the user needed to make on the page. The fact that they needed to choose both a task type and article associated with that task type was not immediately obvious.
 * Descriptions of the task types were not clear to users.
 * The description of the landing page did not make it clear what the process was (i.e. choose a task type, choose an article, edit it).
 * Users definitely read from left to right, implying the randomization of order was a correct choice if want even distribution between selection of different task types.
 * Users found and used the question mark flyouts to help understand the task
 * Some users attempted to click the task icons to select that option.

User experience enhancements:


 * Users obviously expected something like guided tours, which implies it would be valuable for us to build tours for each task flow.
 * The copy in the new Getting Started landing page needs work. The descriptive paragraph should probably be expanded, and the description of each task type should more literal (e.g. "Fix grammar and spelling" instead of "Copyedit").
 * E3 needs to test a version of the landing page that simplifies the choice needing to be made (i.e. not task and article, perhaps not even from multiple options). This should be done either with progressive display or removing the options to choose altogether.

Testing scenario three
We tested a prototype (i.e. a clickable mockup) of two redesigns of the toolbar given to users on GettingStarted task articles. We most wanted to explore whether the visual design fit with user expectations about a progress bar and other elements we added. We additionally wanted to get feedback about whether the request to complete five edits fit the mental model users had about the GettingStarted process.

Our test script was as follows:

For this test, you'll be looking at a prototype (i.e. a preliminary version) of a new interface for Wikipedia. The purpose of this interface is to suggest ways to improve the content. Imagine you just created your account, and you are interested in learning how to contribute to Wikipedia. For the duration of the test, please move your UserTesting.com panel to the bottom left of the screen to make sure it's not obstructing any content. Since this is a prototype, not all links or other functionality will work. If something seems broken, don't worry, just tell us what you expect it should do.


 * 1) Imagine you are new to editing Wikipedia and want to help with simple typo fixes. This page is what you would see immediately after creating your account. What do you think this page is asking you to do?
 * 2) Go ahead and choose the option to "Fix Spelling & Grammar", if you haven't already.
 * 3) Can you identify and fix a typo in the article? For the purpose of the test, you don't need to actually type in a change, just go through the steps you think would be necessary to complete one.
 * 4) If you've completed the steps to fix a typo in the article, is this what you expected to see after? What do you think you're being asked to do next? Please describe what you see in detail after saving, and what you think would happen if you click on any of the links at the top bar.
 * 5) Did you noticed the progress bar at the top of the article page? Please describe what you think you need to do in order to complete the progress bar.

Test group A
With this testing group, we had them try out one of the two mockups we've made for updating the GettingStarted toolbar on articles.

Test group B
With this testing group, we had them try out one of the two mockups we've made for updating the GettingStarted toolbar on articles.

Conclusions
The following are notes about the UX questions we were attempting address in these tests, including solutions.

Landing page

 * Getting Started initial options are clear to users and what they find after selecting a task meets their expectations.

Entry point for editing

 * The edit tab is hard to discover. For many users it gets lost into the chrome.
 * The edit link from the template is frequently used as the main entry point for editing. This might partially be due to the fact that the template appearance differs from production (it lacks orange borders and other styles that might distract from the links).


 * Improvements
 * Show the help guider initially by default will help to locate the way to edit.
 * The 2 extra steps are probably understandable from the user perspective in a “getting started” process.
 * There is probably no need for a persistent help button. The problem seems to be more discoverability than learnability. Once the user discovers where the entry point for edit is, has no problems later.

Discoverability of the toolbar

 * Users focus on the content area during and after editing. This makes the top bar to go unnoticed.
 * The current implementation seems to work better than the prototype since the bar appears some seconds later and makes the page to move, making it more noticeable.
 * The blue bar version seems slightly more discoverable than the green one, but still users don’t look for it until they are asked about the next steps.


 * Ideas for improving
 * Appropriate animations, including loading of the toolbar, animation of the progress indicator after load, or highlighting of the Try Another button.
 * When the initial modal dialog is shown, we have an opportunity to make the user more aware of the toolbar.
 * For the green version, the current step indicator can turn green progressively or use a transition to call user attention.
 * Improvements described in next section will also help for general discoverability.

Understanding the process

 * Users that do not discover the top bar don’t realise they are in a 5-edit process.
 * When users identify the top bar, they understand they are involved in a process but it is not clear what is the process about.
 * Providing a “Make four more edits to complete the editor training” helped in the “blue bar” version to inform users were in “tutorial process for editing”.
 * Users tend to associate work units with different articles.
 * The users that went through the 2nd edit are able to keep going forward.


 * Ideas for improving
 * Provide context on the first steps. Add “Make four more edits to complete the editor training” to the green bar version.
 * Reinforce the “next article” mental model by highlighting this option after first edit. This also helps with the discovery of the toolbar.
 * Additional edits on the same article can also show progress, but we will be encouraging the behaviour that seems more natural to users.

Understanding what happens after completing the process

 * Users don’t had a clear idea of what to expect after the 5-edit process, but there is a notion of “going to a next level”.
 * Leading them to a different and more advanced task seems to fit in their expectations.
 * Becoming a more advanced type of user may be also expected.


 * Ideas for improving
 * The initial getting started page can reflect the tasks the user completed, and add some more types of tasks as other tasks are completed.
 * One of the more advanced tasks can be “dealing with templates”, where the user is asked to improve an article and remove the template when done.