Onboarding new Wikipedians/user testing

We conducted three remote user tests.

Testing scenario one
We asked users to complete the following tasks.


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer. Email is optional.)
 * 2) Take a moment to scan this page [Ed. Special:GettingStarted]. What do you think this page is asking you to do? What do you expect to happen next if you click on any of the links?
 * 3) Follow the instructions provided to get started with editing. Take no more than 10 minutes. If you get stuck, don't worry. We're looking to learn how to make this experience easy for people, so detailed feedback really helps, even if you aren't sure what to do.
 * 4) If you're finished, close your browser tab or window. Imagine you might be meeting some friends for dinner, and want to end your Wikipedia session.
 * 5) Now let's say you're back from dinner and want to go back to editing Wikipedia. Open a new browser tab or window, and return to Wikipedia. Please talk through the process. If you're not logged in, please log in again.
 * 6) If you completed an edit in step #4, where would you expect to find this edit?

Test A



 * Did you edit Wikipedia before this test, even once?: No, I have never edited on Wikipedia.


 * What frustrated you the most? What improvements would have made the process easier?: The editor was very stratightforward, however it took me a few moments to see the 'edit' tab, top right. If it were easier to see I would have more quickly understood how to navigate between regular layout and edit.


 * What did you like about the process, if anything?: I liked the editor. It could have been much more imposing and intimidating, but the layout was friendly and easy to understand.

Test B



 * Did you edit Wikipedia before this test, even once?: No, never.


 * What frustrated you the most? What improvements would have made the process easier?: I didn't see anything that I would really consider instructions. There were notes but nothing to walk me through or really explain anything about the process. I would want more step by step 'how to' instructions.


 * What did you like about the process, if anything?: I like that you can edit these pages, but other than that I didn't like much because everything seemed so difficult and too big project.

Test C



 * Did you edit Wikipedia before this test, even once?: No, I did not edit Wikipedia before this test, not once.


 * What frustrated you the most? What improvements would have made the process easier?: What frustrated me the most was not being able to go back and locate my edit. I don't feel the editing process was that hard, just finding what I edited was difficult. I would improve this by adding a button to see what you've edited in the past, that would make it easier.


 * What did you like about the process, if anything?: What I like about the process was the simplicity of it. It seemed pretty easy to do the actual editing.

Conclusions

 * We specifically asked the testers what they thought Special:GettingStarted was asking them to do, and what would happen if they clicked on any of the links. All of the users understood the landing page correctly, and the request to choose a page and contribute to it.
 * While users understood the GettingStarted interface was asking them to edit, they assumed that they would have to have some prior knowledge/expertise in the subject of the articles in the list.
 * There was clearly not enough instruction about the kind of task requested and how to complete it when users were on an article they chose from the list.
 * Users had less difficulty with wikitext than one might expect, almost certainly due to the nature of the task we were asking them to consider. They generally ignored markup that was irrelevant to their work, and some even identified sections within articles. The smaller maximum length of onboarding articles, 10k bytes, may be a factor.
 * None of the users completed an edit during the test, though 2/3 of them made it to the edit screen, then described precisely what they would do. (The one user who attempted to actually save an edit ran in to the CAPTCHA delivered to users when they save an edit with an external link addition.)
 * Users all navigated to Wikipedia through the main wikipedia.org portal, using either the link to English Wikipedia or the search function to find "their version".
 * There are a number of other insights in to how users learn to navigate Wikipedia, including key functions such as search, history pages, watchlists, and more. Those processes are currently outside the scope of this project, so we won't elaborate on them yet.

Test scenario two
This was a test of the new Special:GettingStarted landing page, sans guided tours, launched on Thursday, March 7th. We asked users to complete the following tasks:

"This is a test of the signup process for Wikipedia and a new interface for getting started as an editor of the site. In this scenario, imagine you've decided to join Wikipedia to update and improve its information. Remember, we're testing the interface, not you. If you're having difficulty with something, the problem is with our design. Please "think out loud" as much as possible.


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer.)
 * 2) Take a moment to scan this page. What do you think this page is asking you to do? If you want to become a Wikipedia editor, what would you do next?
 * 3) Once you've scanned the page, go ahead and try to make your first contribution as a Wikipedia editor. If you get stuck, don't worry! Just explain what you would do next and why. It's okay to say you feel like you would give up at any point."

Test A



 * Did you edit Wikipedia before this test, even once?: I've done it before, but never consciously being aware that I was being a copyeditor perse, more just clicking on edit on the top when I've noticed something needs to be hyperlinked or spell-corrected. Probably done it over a hundred times by now.


 * What frustrated you the most? What improvements would have made the process easier?: You might explain the process with some kind of information or something on that 2nd page that one comes to after signing up to sord've let them know what the next step is, that page kind've leaves it up to someone to guess what to do next.


 * What did you like about the process, if anything?: I liked editing the errors using the edit feature, works really well. The signup process was easy too.

Test B



 * Did you edit Wikipedia before this test, even once?: I have never done it before today.


 * What frustrated you the most? What improvements would have made the process easier?: Reading the strange, foreign format confused me the most. That would take some getting used to. I would suggest the editor read the article first and then go to edit. I would also show some examples of what is importatnto be edited- that would help.


 * What did you like about the process, if anything?: It is fun to improve things- I liked reading and revising.

Conclusions
Issues identified from the tests:


 * In the first two tests, we unintentionally primed users to choose the "Copyedit" task by using the terminology "Wikipedia editor". This was corrected in a third user test script.
 * There was confusion about what kind of decision the user needed to make on the page. The fact that they needed to choose both a task type and article associated with that task type was not immediately obvious.
 * Descriptions of the task types were not clear to users.
 * The description of the landing page did not make it clear what the process was (i.e. choose a task type, choose an article, edit it).
 * Users definitely read from left to right, implying the randomization of order was a correct choice if want even distribution between selection of different task types.
 * Users found and used the question mark flyouts to help understand the task
 * Some users attempted to click the task icons to select that option.

User experience enhancements:


 * Users obviously expected something like guided tours, which implies it would be valuable for us to build tours for each task flow.
 * The copy in the new Getting Started landing page needs work. The descriptive paragraph should probably be expanded, and the description of each task type should more literal (e.g. "Fix grammar and spelling" instead of "Copyedit").
 * E3 needs to test a version of the landing page that simplifies the choice needing to be made (i.e. not task and article, perhaps not even from multiple options). This should be done either with progressive display or removing the options to choose altogether.