Talk pages consultation 2019/New user tests

The talk pages consultation set out to gather perspectives on communication from as many parts of the Wikimedia community as possible. For that reason, a lot of outreach was done to specific wikis and in many different languages, which gathered perspectives from many experienced Wikimedians. But it is also important to incorporate the perspectives of newcomers -- in other words, the future Wikimedians who aren't yet part of our communities, but that we hope will join us one day. Those Wikimedians may come from different cultures and have different expectations of technology than existing Wikimedians, and we don't want the current state of talk pages to keep them away.

In order to try to understand how new users feel about the current state of communication on wiki, we used UserTesting.com. UserTesting.com manages panels of people who receive compensation for recording themselves while testing software and talking aloud about their thoughts. For this test, we recruited five people who have never participated in wiki discussions to record themselves trying out real talk pages for the first time.

Goals

 * 1) Test the discoverability of talk pages: how hard is it to even find the place to have a discussion?
 * 2) Test the usability of talk pages: how hard is it to reply in conversations using the correct formatting, and to start new conversations?
 * 3) Test the expectations for after they use a talk page: when will they get a response, who will respond, and how will they find out?
 * 4) Ask users how communication could be improved: what would they change about the experience they had?

Choosing the testers
We wanted our testers to reflect the sort of people who would be likely to encounter talk pages. That would mean a certain amount of technical literacy, familiarity with Wikipedia, and to be someone who might want to edit. To narrow to those people, we asked a series of screening questions, such as "How often do you look something up on Wikipedia?", "Have you ever engaged in a discussion with other users on Wikipedia?" and "If you have not edited Wikipedia in the past, what would you say is the main reason why you have not edited?" Only if users gave all the answers we were looking for were they allowed to take the test.

All five of the users in this test used desktop Wikipedia in English. It is possible to run tests on mobile, and in other languages in the future.

Test procedure
The test was centered around a copy of the Mountaineering article from English Wikipedia. Users were prompted to go through multiple steps and talk aloud about the experience. A summary is below.


 * 1) Make a small edit to the copied article itself, to make sure the user understands how editing works in general.
 * 2) Talk about how they would find the discussion area.
 * 3) Go over to the talk page and explain what they are seeing, including WikiProject templates at the top.
 * 4) Add a comment to an existing conversation, with prompts to figure out how to indent and sign their comments.
 * 5) Add a new topic to the talk page with a signature.
 * 6) Talk about their expectations for when they will get a response and how they will find out about their response.

Results and quotations
Below are aggregated notes from the five interviews, along with some select quotations in italics directly from the user tests.


 * Overall
 * None of the five testers expressed surprise or indignation at the way on-wiki communication works. That said, they were all confused at times, and all suggested a general modernization. "It would be nice if it was more like an actual message board.  A bit more simple."
 * All the users generally expressed that on-wiki communication felt "old". "It seems more old school and html-like. It's fine but it's definitely old fashioned."
 * There are many unintuitive elements of talk pages that users can figure out after some experimentation. An important question is whether those initial moments of confusion drive potentially good contributors away. "It will bring me to a web-based forum full of threads.  But this is not at all what I thought it was going to be.  That's kind of weird, isn't it?"
 * Discoverability
 * All users struggled to find the talk page. Most thought that clicking "edit source" in an article section would lead them to a discussion forum about that section.  None of them found the "Discussion" tab at the top of the page.  This may show an expectation that users have about being able to discuss specific sections of an article, instead of article as a whole.  "Yeah, I guess I would click 'edit source' again, because I don't really see any other place I could go."
 * Similarly to the point above, once on the talk page, many users thought the headers of the discussions corresponded to the sections of the article. In other words, they thought that discussions about articles were organized around article sections.
 * After looking at the WikiProject boxes at the top of the talk page, no users correctly intuited what a “WikiProject” is, and none of them gave a good explanation for why it would be on the talk page. "The tan boxes are telling me that the information from this page can also be used within those other two pages."
 * To describe talk pages, they all used phrases like "message board" or "discussion forum", and assumed that there would be threaded discussions like on other websites they are used to.
 * Usability
 * Some users thought that the correct way to join a discussion would be by clicking the "talk" link in a user's signature. They thought that was like a "reply" link.  "I think I would click on talk, this button right here."
 * All users perceived that they would join the discussion by typing their comment at the bottom of the section.
 * All users realized that using colons would indent their reply. They realized this because the test had them reply to a conversation that already had several indentations.  It seemed unlikely that they would figure out how to indent without having an example to follow.
 * Two of five users were able to figure out on their own how to sign their comment. One did this by clicking the signature button in the editing toolbar, and the other did this by noticing the "use four tildes" prompt that is present on English Wikipedia talk pages.  The other users intended to sign by just typing their user names.  "Okay, so I didn't even know you could do that."
 * All users commented that it would be better if signatures and indentation were done automatically. "Be a bit more up-to-date with easy one-click access to reply and add signatures. Use verbiage like 'click to discuss' instead of 'edit source'"
 * Expectations for after posting
 * All users perceived that their comments were in public and would be seen by anyone who visited the page.
 * Users understood that the time that it would take to receive a response varies by how busy the page is.
 * Many users expected to find out about replies either through their on-wiki notifications or through email.
 * Suggestions for improvement
 * All users commented that it would be better if signatures and indentation were done automatically. "I feel like it should automatically indent them.  I don't feel like I should have to do the colons in order to do that.  And I feel like there should be an easier way to add your signature at the end, because I don't feel I would have known how to do that without doing this user test.  Otherwise I think it was fairly easy to reply."
 * Users commented about how it should be easier to find the talk page.
 * Several users said that this format felt "old".
 * Other interesting moments from the tests
 * One user wasn't sure whether "edit source" on the talk page would open the editor for the talk page or for the article. "I don't know if that actually will edit the source inside the [section on the page], or you can actually talk inside of here."
 * One user expected that clicking on someone's username would show what other discussions the user is involved with. "Maybe we could click on them and see what other conversations they're involved with."
 * One user wasn't sure whether newer comments were at the top or the bottom of a thread.
 * One user wasn't sure whether they would be alerted if their comment was answered. It shows that there is potentially some work to be done to orient users when they are first engaging in discussions so they know what to expect.

Next steps
Having completed these tests, there are a few other elements we want to learn more about in some future tests. The tests done so far use minimal WikiProject templates at the top of the talk page that are only a couple lines. We want to see how much more confusing it is when a talk page has much more non-talk content on it, like extensive WikiProject templates, warning templates, instruction templates, and archives.