Flow/Research

Research on talk discussions has been a long-running process.

Past research on user talk






"Ignored" new users
Since 2006, new users who are "ignored" (e.g., do not receive any talk page messages) are more likely to keep editing than users who do receive a talk page message. This suggests that the current way we communicate to new users is doing more harm than good.

However, for new users whose edits are reverted, not sending a warning is much more likely to make the user go away forever than sending a template warning. 41% of new users will continue editing after being sent a template warning for a test edit, versus 9% who will continue editing if reverted and sent no warning.


 * Documentation (1, 2)

First messages to new users
Since 2006, English Wikipedia has seen an explosion of semi-automated tools and bots that send welcome and warning messages to new users. By 2011, 80% of first messages to new users were templates sent by a tool or a bot, and 65% were warning templates.


 * Documentation

The impact of user talk templates on new users
About 50% of registered and unregistered users check their talk pages within 24 hours of receiving a message. The majority of users whose edits are reverted and who receive a warning never return to edit again. Changing the tone of template warnings has a small but significant effect on short-term retention, but not long-term retention.

Documentation

New user help requests on user talk
Based on a sample of new user edits from 2011, the majority of new editors ask for help on user talk pages – their own or another editor's.


 * Documentation

New User Test Videos
The Foundation conducted several user tests to get a glimpse into the problems facing new users when they are exposed to what are relatively simple (and common) scenarios with user talk pages.

Note that the links to these tests leave the Foundation website and go to an external site, usertesting.com, which may have a different privacy policy than the Wikimedia Foundation's.

Tests have been annotated, calling out specific issues that users face and behaviors that they expect.

Descriptions of the individual test may be found at FlowPortal/Research/User tests.

Test 1
This test was a very simple task for new users who have made an edit and then were reverted. We tested to see if the user was able to see that they had new messages and were able to successfully reply to the messages. The page was a very simple and common scenario.


 * Aeldar02
 * User expected a "messages" section in user tools area.
 * User wishes actual messages were called out; does not understand that "Welcome Template" is also a message (there's no distinctions)
 * User has difficulty figuring out how to reply to messages on their own talk page
 * User has extreme difficulty figuring out how to leave a new message for another user
 * User assumes that "watch this page" will send them a "read receipt". User wants to know that their message has been sent. She somehow ends up on page history trying to see if her message was sent, and is completely overwhelmed.


 * Online Bee
 * User has difficulty understanding how to reply to messages (not sure where to reply and does not understand that original messages and replies may not be in the same place)
 * User actually signs a post, but does so with cut and paste - and still types her name, not knowing that the tilde codes expand
 * User isn't sure that message was sent, nor where it went.
 * User reopens message section; sees wikitext; assumes it is to edit original message and not respond (totally does not get colon indentation)


 * Paul
 * Does not understand yellow message banner. Takes a long time to figure this out.
 * Does not understand where messages begin and end
 * Accidentally clicks on "last changes" and is sent to Diff-land. Whupz.
 * Confused by a lack of a "reply" button
 * When he finally gets in, he puts the response above the section as if he were replying to an email.
 * Cuts and pastes tildes for response (this appears to be a keyboard problem)
 * Gets completely confused; adds a template, which triggers a captcha.


 * Test Video not available (This user declined permission to share the video, but takeaways are included below)
 * Successfully locates new messages.
 * Gets lost by the edit screen
 * Assumes welcome message and the revert warning as a single message.
 * User does not respond on her talk page; instead goes to the other user's talk page.
 * Tries to leave a message there; gets confused. Does not see a way to reply.
 * User expresses confusion about the difference between responding to a section and starting a new section
 * User starts new section in response to original message, does not include a summary. Upon save, this results in confusion (she cannot find the message she left). Assumes there must be a delay between saving and delivery.
 * User tries again, this time adding a subject (thinking that the missing subject may be why it didn't appear in the first place). She correctly inserts her signature. However, this is because she see charinsert tools (the whole editor does not fit on her screen).
 * User ultimately gives up in frustration (this is probably why she did not give permission to share the video).

Test 1.1
This test was the same as the first except for a minor clarification in one of the tasks (referring to a specific edit to "Dwarf cat" over generically described "edits").


 * GlobalE
 * User assumes that there will be a "messages" link in the user tools section
 * User thinks his talk page is a "welcome" page (given the welcome template left)
 * User assumes talk page will also include his contributions
 * User does not think the welcome template is a message.
 * User correctly gets the edit link
 * User has several issues with the editor and its usability
 * There is a Kafka-esque journey into existential torment as the user believes that he must cut and past wikicode for an icon in front of his message for it to be seen as a message.
 * User responds at the top of the message (like an email), and the section subject becomes part of his signature
 * User thinks a namecheck/mentioning system would be great ("It would be great if it worked like twitter where you can add an @ sign")
 * So he tries that, cutting and pasting their username wikicode.

Test 2
This test involves a more complex scenario, wherein the user has asked for help from another user. It involves a "talk back" template and introduces the concept of minor threading.


 * Kayaker
 * User is confused by "new messages" and "last change" links in the orange bar
 * User thinks Welcome template is automated
 * User actually understands that displayed indentation means a threaded conversation
 * A major issues is that this user does not understand that the talk page contains several converations
 * User encounters the phrase "talk page stalker"; thinks its funny, but uses that knowledge to understand why a third party would get involved (suboptimal, but interesting)
 * User expects a "reply" link in signatures but eventually clicks edit on the section after some confusion.
 * POW. User is completely confused by a giant block of wikitext.  Has many minor hiccups with the editor.
 * "I don't know where to begin." Completely confused by colon indentation; assumes that you need to know web programming; thinks wikitext is HTML. Is afraid that she'll screw up the entire page if she types anything.
 * "This is very intimidating to me"
 * Unsure which bits are manual and automatically inserted (e.g., dates, etc.)
 * User makes connection between colons and threading, but cuts and pastes them rather than types them.
 * User has significant negative experience with wikitext blocks. Expresses extreme frustration and dislike.
 * User does not understand that someone else would need to be "poked" to get a reply but does have a glimmer of understanding regarding "watch this page"
 * User is suprised that they may have actually succeeded.


 * kawzx7
 * The user does not understand signatures nor intiuitively understand indentation
 * User figures out signature code after some difficulty
 * Does not understand talkback templates are left by hand; assumes that the software will do it for him.
 * Confused that talkback templates are not auto-updated with newer timestamps.
 * Totally surprised that he can edit someone else's talk page (even though he did it earlier). Does not think he should or that he should even have the ability to do so.
 * User leaves reply under the wrong location (does not get talk back templates at all)
 * User deletes the talkback template

Test 2.1
This test is the same as the previous, only fixing a small bug in the text of the test.


 * PJensen66
 * User attempts to use search box to find their messages; fails. Eventually finds new messages bar. Feels stupid for not seeing it earlier.
 * User is obviously confused by the talk page and how to use it. Ends up in several different locations (same user talk page twice, then a user page, then their own talk page, then back to the first user talk page). Discovers a flaw with user pages in general (no "leave a message" link by default).
 * "I don't know how to put a response in here."
 * User drops in and out of editing the section several times, confused by wikitext.
 * User gives up on responding to the messages. Is forced to move forward with the task.
 * User cannot find the question they are responding to within the wikitext block.
 * User realizes that they didn't get a signature on their response.
 * User cannot understand talkback templates at all and ends up on help pages trying to figure them out.
 * "At this point I'd pretty much give up on Wikipedia and I wouldn't bother."
 * User spends a great deal of time reading help pages about Talkback templates.
 * "I'm just kind of feeling like an idiot now."
 * User gives up. Declares that he would not waste his time trying to figure the system out.

User Test Takeways

 * None of the tested users were able to intuitively grasp anything about the use of User_talk pages.
 * On average, it takes new users around 15 minutes to grasp the basics of user talk messaging.
 * Users generally have negative impressions of using wikitext for communcation ("it's like math"). It is extensively confusing and somewhat demoralizing.
 * Users have expectations that the discussion system will be significantly more modern and be much more user friendly. To wit:
 * Users have difficulty locating their new messages and talk pages.
 * Signatures and colon-indentation are especially confusing.
 * Users expect an obvious mechanism for replying to comments. They do not understand section edit links.
 * Users do not generally understand how to watch for replies (or if this will even happen).
 * Users have an expectation that those they engage with will be notified in a sane manner.
 * Users expect a way to leave messages for other users from User pages (not just User_talk).
 * Talkback templates are completely confusing and non-intuitive.
 * Users are surprised that they can edit other people's messages.
 * Users have difficulty understanding that conversations can exist in multiple locations.

Note that the scenarios that these users are sent through - and the talk pages they end up on - are typical experiences for new users.

Experienced Users: Quora Question
User:Jorm (WMF) asked on Quora: What do you find confusing about Wikipedia's discussion systems?. Here is the question and answers.

What do you find confusing about Wikipedia's discussion systems?
Not everyone is aware that there actually *is* a discussion system on Wikipedia. There are two basic types: discussions about articles and then per-user discussion spaces (these are called "Talk pages").

I am currently working on a design proposal to build a better discussion system. I know what I think is confusing about Talk pages (there's a big list) but I want to hear what other people think, too, so that I can fold issues I may have missed into the design.

Pete Forsyth
Pete moved his answer from Quora to mediawiki.org

Since perspective is important on this, here's where I'm coming from: I'm a long time (8 year) heavy user of MediaWiki (many different sites), most notably English Wikipedia; and I do a great deal of work with first-time and developing wiki contributors. So, my comments reflect both my own needs, and my perception of the needs of the people I teach.

Things that don't get discussed much, but are big opportunities for improvement:
 * Links (to talk pages, to section headers of talk pages) do not persist through through archiving. Asynchronous, eventualist communication is one of the core strengths of wiki software; this is a major exception. (e.g., I link today to a conversation -- from Twitter, say -- but next month, that link goes nowhere useful.)
 * It's difficult to provide context. Outside Wikipedia, most of my collaborative writing is in Google Docs; and the "comment" feature, where you highlight exactly the chunk of text you are talking about, is incredibly useful. I find myself missing the ability to easily point others to exactly what I'm talking about frustrating.
 * There's no easy way to "search ahead" when typing somebody's username or other internal link. The HotCat extension has this capability; on Twitter and Facebook, when I type "@" followed by the first few letters of a friend's name, it starts to guess what I mean. A feature like this in MediaWiki would be hugely helpful.
 * No "ping" notification. If somebody tags me in Facebook, I am notified; if somebody links my name in a Wikipedia discussion, there's no notification capability.
 * Posting diffs can be a really important element of descriptive or persuasive discussion. A quicker and/or more intuitive way to find and paste a diff would be a big help.

Obvious/well-known issues with perceiving what is going on:
 * When somebody comments on my user talk, do I reply on my own page or on theirs? There are pros and cons to both.
 * When somebody replies to me, I'd like to be notified. "Watch list" is too  general, expecting them to tell me on my own user talk is too onerous.
 * Same as #2, but cross-wiki: multiple languages, multiple projects.
 * Too-specific time stamps result in a whole lot of clutter. It's very, very rare that information about what precise time a comment was left is important to the discussion; if that information is preserved a click away (e.g. in edit history), it could be left out of the display in the context of discussion. Perhaps dynamically: for instance, anything more than 3 hours old suppresses the minutes and seconds, anything than two days old suppresses the time of day.

Obvious/well known issues with adding comments:
 * I shouldn't have to think about whether/how much I am indenting…much less need to use markup to do so.
 * I shouldn't have to remember to sign.

Waldir
This was a reply to Pete's comment on mediawiki.org

This includes most of the points that annoy me about the current discussion "system" in mediawiki. There are a few things that I would add, though:
 * Thread-independent management. Have each thread be a single entity would allow watchlisting only the topics you care about, moving threads to more appropriate pages, and archiving without losing edit history, or, if archival is done via moving the page to preserve the edit history, without having to split the page at unnatural places (e.g. if organizing archives by year, the first posts of a still active thread near the end of the year will have to be copy-pasted back to the current page, losing history; if organizing archives by number of threads, reactivating a prematurely archived thread would require manual copy-paste and loss of history)
 * Native support for quoting. This is especially relevant for long messages, since inserting replies below parts of the original post break its flow and make it unclear who is writing what, and quoting templates are a hack that can't, for instance, link reliably to the original post being quoted.
 * Features for consensus-making. Wikipedia discussions can get quite big and it would be great to have a way to provide summaries for threads, and voting for individual posts, so people who come across a long discussion don't have to read the whole thing to make sure they're not repeating a point already made (and potentially addressed/resolved), or overlooking important points that should be taken into account. Voting would also make it easier to ignore low-vote posts if one's short on time, and allowing the best posts to be highlighted would even allow some form of summarizing to the thread.
 * Automatic signatures. Not only the tildes markup is unintuitive, it's also very easy to forget to sign, even for experienced editors.

Joseph Fox
I would say the main issues are that it is too difficult to understand the "indenting" feature we use to thread conversations - I can't imagine a new user just guessing that that's what the colons do; the idea of four tildes to sign a comment seems a little strange given that on most sites with discussion features (Facebook, Twitter, Disqus) the user's handle is inserted by default; and it is much too easy to accidentally distort the fabric of what someone else has already said, and then of course be vilified for refactoring comments.

Perhaps I'm well off the mark given I've used Wikipedia and its talkpage format for almost seven years(!).

Dan Leveille
(Editor since 2006)

Discussion on Wikipedia is very unintuitive and has been frustrating to me. Glad to see that it's getting improvements!


 * It's a wiki rather than a discussion system. Wiki markup is confusing, especially for new users. As someone with HTML and programming knowledge, I always ask myself, how would non-programmers understand this?
 * Formatting. Some people indent to start discussions, some people do not indent properly, and sometimes it's hard to follow the conversations. Sometimes people will add their comments out of order.
 * It's unclear how to start a new discussion. For new users, it's unclear how to start a new discussion topic. Sometimes people will just add it to the top or bottom of the page without a heading.
 * Archiving discussion. Discussions get moved to archive pages, but this system is less than ideal and isn't entirely intuitive.
 * It's labeled "Talk". This isn't all that intuitive, and could imply that it's a discussion forum about the topic rather than the actual article.
 * People use it as a discussion forum about the topic. (WP:NOTFORUM)
 * You have to sign your comments. You have to type " ~ " to sign a comment. How would new users know this? I always used to forget. Unsigned comments sometimes get signed by a bot, which is confusing to new users and adds extra clutter.
 * There are no profile photos. Discussion is different when you're talking to a person rather than a username. If you see someone's face when you're talking to them, you will act and talk differently. Less anonymity makes people think more about what they're saying and would add more of a community and personal feel to Wikipedia.
 * Talk Page Notices add clutter. There's sometimes a ton of notices at the top of the Talk Page. It gets confusing and cluttered, and pushes discussions way down off the page. Not only that, but they have different looks and design styles, and it looks really messy.
 * Notifications. You don't get notificiations when your comment or discussion gets replied to, and you don't get notifications when new threads are started (unless you're watching the Talk page and happen to see that it was updated).
 * Subscribing to a thread. There's no way to "subscribe" to a specific conversation on a Talk Page. It would be awesome to see the number of subscribers to a conversation as well, to show how "hot" or "important" the discussion might be.
 * Upvoting. It would be great if people could upvote comments to show their support for points others make.
 * Proposing article merges or deletion gets confusing. I have to research how to do it, then place these templates on both articles and both Talk pages, and then choose one Talk page to discuss it on, then point to a Talk section about the merge? You definitely need to have friction, so that not everyone will do it, but right now, it's not the right type of friction. It just feels like a mess and overly complicated. I always feel like I might not be doing it right.

Erlan Vega
(Wikipedia editor since 2005 on eswiki)

I found that using the discussion is easy and intuitive. However, not many people do so. About user talk pages for example. Some pro users started answering posts on their talk pages rendering the notification system mute. To solve that, a bot runs or the user places a template saying that they will not answer in your talk page. Wikipedia is the only place where I know conversations are split in two pages, but, extrangely, it has worked for the most of us. I should say, though, that good people didn't stay because that was confusing and told me they expected something like facebook message system.

As for the talk pages for articles, if they ever get messages, few are answered and you don't get that nice orange notification you get when someone writes on your talk page. You somehow have to figure out that you have to place that page on your watchlist and then read it from time to time to see if someone, by chance, answered to your comment.

What's most frustrating I think is that some power users blame newcomers for not using a feature that isn't obvious to everyone.

Mark Savchuck
I had no idea it existed... And I use Wikipedia every day. So yeah...

I guess it doesn't stand out enough.

Also - maybe you can run the same thing you do when you fundraising? - Make a bright box on top that has a video explaining the feature and how to use it. Run it for a month (most of the active users will see it by then) and we will start using it more.

Thank you for the feature (because when I read Wikipedia, I usually have questions, and there is no way I can ask someone....).

Fabien Benetou
It is "too free".

People are used, by order of popularity, to
 * 1) Face-to-face dialogue
 * 2) Phone dialogue
 * 3) Linear forums

Basically anything slightly out of those formats are very hard to use because we lack familiarity.

Wikis are amazing because they provide an incredible freedom to focus on the content first.

Unfortunately discussions are as much about the content as about the social aspect.

I suppose then that successful platforms for discussions online lower cost to track "who said what and in what context".

For that I suggest following most guidelines from forums, but adapted to efficient Wikipedia editions


 * Making signing, ~, automatic
 * Facilitate quotation to the specific problematic part of a text (wiki page or discussion part)
 * "Suggested edits" a la Quora

Experienced User Suggestions
These are suggestions from experienced users for improving the user-to-user discussion system. This list is by no means exhaustive, and is the result of a limited set of answers.

Bugs/Issues

 * Too much wikitext:
 * Signatures should be automatic
 * Indenting is tedious
 * Managing these errors by inexperienced users adds workload to experienced ones
 * Wikitext is "too free"
 * The label "talk" is counterintuitive.
 * It doesn't stand out enough that there is a conversation space.
 * No way to watch specific threads; must watch entire page.
 * Archiving is completely broken in many ways.
 * Lack of permalinks is especially bad
 * Context and history in conversations is typically lost
 * Where do replies go? My talk page or someone elses?
 * Lack of notification when someone mentions you in a discussion or replies to a discussion you are involved in
 * Timestamps can be difficult to follow or clutterish
 * Thread/topic management is an issue.
 * To easy to distort conversations through refactoring
 * Talkback templates/notices add clutter
 * On-wiki processes (such as suggesting merges or deletions) are complex and difficult to follow

Feature Requests

 * A desire for context/annotations
 * Namechecking/mentions (@Username)
 * Better way of linking to diffs
 * Native support for quoting
 * Profile photos or avatars. Discussions can be hard to follow and lack empathy
 * A way to "upvote" comments or indicate simple gratitude or support