Onboarding new Wikipedians/user testing

We conducted three remote user tests.

Background
We asked users to complete the following tasks.


 * 1) Create a new account on Wikipedia. (Use any name and password you prefer. Email is optional.)
 * 2) Take a moment to scan this page [Ed. Special:GettingStarted]. What do you think this page is asking you to do? What do you expect to happen next if you click on any of the links?
 * 3) Follow the instructions provided to get started with editing. Take no more than 10 minutes. If you get stuck, don't worry. We're looking to learn how to make this experience easy for people, so detailed feedback really helps, even if you aren't sure what to do.
 * 4) If you're finished, close your browser tab or window. Imagine you might be meeting some friends for dinner, and want to end your Wikipedia session.
 * 5) Now let's say you're back from dinner and want to go back to editing Wikipedia. Open a new browser tab or window, and return to Wikipedia. Please talk through the process. If you're not logged in, please log in again.
 * 6) If you completed an edit in step #4, where would you expect to find this edit?

Test one



 * Did you edit Wikipedia before this test, even once?: No, I have never edited on Wikipedia.


 * What frustrated you the most? What improvements would have made the process easier?: The editor was very stratightforward, however it took me a few moments to see the 'edit' tab, top right. If it were easier to see I would have more quickly understood how to navigate between regular layout and edit.


 * What did you like about the process, if anything?: I liked the editor. It could have been much more imposing and intimidating, but the layout was friendly and easy to understand.

Test two



 * Did you edit Wikipedia before this test, even once?: No, never.


 * What frustrated you the most? What improvements would have made the process easier?: I didn't see anything that I would really consider instructions. There were notes but nothing to walk me through or really explain anything about the process. I would want more step by step 'how to' instructions.


 * What did you like about the process, if anything?: I like that you can edit these pages, but other than that I didn't like much because everything seemed so difficult and too big project.

Test three



 * Did you edit Wikipedia before this test, even once?: No, I did not edit Wikipedia before this test, not once.


 * What frustrated you the most? What improvements would have made the process easier?: What frustrated me the most was not being able to go back and locate my edit. I don't feel the editing process was that hard, just finding what I edited was difficult. I would improve this by adding a button to see what you've edited in the past, that would make it easier.


 * What did you like about the process, if anything?: What I like about the process was the simplicity of it. It seemed pretty easy to do the actual editing.

Conclusions

 * We specifically asked the testers what they thought Special:GettingStarted was asking them to do, and what would happen if they clicked on any of the links. All of the users understood the landing page correctly, and the request to choose a page and contribute to it.
 * While users understood the the getting started interface was asking them to edit, they assumed that they would have to have some prior knowledge/expertise in the subject of the articles in the list.
 * There was clearly not enough instruction about the kind of task requested and how to complete it when users were on an article they chose from the list.
 * Users had less difficulty with wikitext than one might expect, almost certainly due to the nature of the task we were asking them to consider. They generally ignored markup that was irrelevant to their work, and some even identified sections within articles. The smaller maximum length of onboarding articles, 10k bytes, may be a factor.
 * None of the users completed an edit during the test, though 2/3 of them made it to the edit screen, then described precisely what they would do. (The one user who attempted to actually save an edit ran in to the CAPTCHA delivered to users when they save an edit with an external link addition.)
 * Users all navigated to Wikipedia through the main wikipedia.org portal, using either the link to English Wikipedia or the search function to find "their version".
 * There are a number of other insights in to how users learn to navigate Wikipedia, including key functions such as search, history pages, watchlists, and more. Those processes are currently outside the scope of this project, so we won't elaborate on them yet.