Quality assessment tools for Wikipedia readers

This is a very loose transcript of the discussion for the session "Quality of Article Content and the Reader Perspective (for Wikipedia)". Nothing is an exact quote but the spirit ofthe main points of what was said shuld have been captured reasonably well.

A: What is the average reader perception of quality when they visit a Wikipedia article? They probably come from Google search, there are of course various templates (Good, Class A) but even if the reader knew what these meant, they are on the discussion page and not visible. The reader doesn't even know the discussion page exists.

G: We should make the discussion page more visible to the reader.

P: The reader won't look at all that text, they want something quick.

G: Do the readers care about the quality of the article? Anyways we would never tag it with something like a star or anything like that.

(Note that some Wikipedia articles are right now tagged as "Good" which displays a green circle with a plus sign in it in the upper right hand corner. This was done by a bot run on May of this year, from the list of articles marked as g"good" using the same template on the talk page.)

A: How do you in this room make the determination that content on the Internet is trustworthy? What standards do you use?

K: It depends what institution is behind the article. For example, if you are looking at health care you would want to look at NHS.

P: Maybe Wikipedia articles are more helpful to the average reader than NHS pages? If you are looking for a general overview of a topic...

C: Wikipedia articles often contain too much information and detail for the layperson.

P: The level of the article is just too difficult to decipher; high school students get told to go to WikiHow or WikiAnswers for a short summary.

G: Should we try to separate readers into classes (i.e. high school, general layperson, etc) for purposes of talking about quality?

C: Chemistry articles for example are just not accessible to the layperson. They have so much specialized information in them.

G: The student doesn't realize that Wikipedia is unreliable; if they did they would be more interested in reading the dicusssion page then.

C: If we want to focus on literacy in a sense, on a deeper understanding, we would want peopel to look at the article history, we want to empower the reader to understand how the article was created. Is there controversy on the talk page? If not, and it's well-written, then we can assume it's probably a good article.

A: What do we mean by "well-written"?

C: Well I think we can recognize it when we see it. Some articles, they have one sentence here, anotehr one there, there's no coherent narrative, maybe pieces are repeated in different parts of the article, no one has gone through the whole thing and unified it. A well-written article has cohesiveness, good structure with defined paragraphs, a flow.

P: Some articles are written in an essay style, while still neutral pov, others are more traditionally encyclopedic. Some are too hyperlinked; maybe lay readers lose the plot?

A: This reminds me of the classic xkcd comic: http://xkcd.com/214/

C: Yeah, that's my experience with Wikipedia a lot of the time too.

P: We need a way to visually represent the links between articles and between areas of knowledge. A visual structure. Britannica did this, at least on DVD. For example, how this medical article connects to these other base disciplines... It's good to have "Further reading" and references, but how can the reader find other resources without some sort of catalog? We want to look at these things from a navigation and epistimological point of view. There are wikis that target specific audiences though, for example Wikikids (for children).

C: I think France has a Wikipedia for children.

G: vikidia.org, which has French and Spanish editions. It's a different project, not based on Wikipedia. I think the content is entirely separate.

P: There's the Encyclopedia of Life, crowdsourced, repurposing of Wikipedia content. They request help from specilized experts. It's a biodiversity archive, and the cooperate with flickr and national history museums and such. It's funded by the McArthur Foundation. They put observations of species on line. They filter and correct Wikipedia content as they re-use it. So we could think about quality of content on a per-subject basis, working with specific communities, collecting expert input.

C: The more different groups with different expertise you have working on the same article the more accessible it will become. If there are too many experts...

G: I think there are two things we have to do, 1) make it possible for readers to realize that they are the ones that must evaluate the article content, and 2) provide them with assessment tools. We could have a quality toolbox. For example they could turn on or off Wikitrust, they could see how many revisions there were in the last week, and so on. No one tool would be enough.

P: There is Diigo, which lets people see what other people said about a page. There is also Cohere, which is an openuniversity web annotation tool. You could let people add a check mark or a star, there could be some sort of widget for that. The reviewing could be networked.

K: Suppose you are looking at a history article, about some king, born in a certian year. How do you as a reader ensure that this is correct? Even with citations people still make mistakes.

G: If you turn on WikiTrust, let's say you see that the article has been very stable over the last 3 years, it's probably good. If you see it's been stable over the last 6 months but the date of birth was updated yesterday that might put that information in doubt. Masybe we could have a collection of tools, a reader could choose theiur favorite one?

C: I like the suite of tools idea. How about a "rate this article" tool?

G: That's going in during the next year.

C: What does a mixed rating mean though? Is there missing information, was the article useless for some purpose?

G: Over the next year the tech department at WMF is going to be involved in more research, looking at what features to develop. The "rate this article" feature is important to the ED and to the Board. There is an existing extension but it may not be so viable. I think the usabillity folks Trevor, Nimish, Paru, maybe Howie? were looking at it. There is user research and testing going on now.

C: Rating doesn't give as richa picture as the discussion page.

G: The toolbox could have, say, the two last discussion topics from the talk page.

P: We don't want to confuse the reader by putting too much stuff there, we want a simple clean interface.

G: Two main principles of usability are to hide anything irrelevant and to make things discoverable. If we put it in the toolbox but it's in there when someone looks at the contents, they can scroll and find it, then that meets both principles.

P: Maybe we could look at something flagged semantically in the article as controversial or check for key words on the talk page (e.g. "neutrality", "policy") and these could go in a little comment box.

C: Maybe we could enrich WikiTrust so the text would have a marker "this was useful to me in this specific context" or "this part was well written", the article could be annotated.

A: Tagging/annotating is a familiar paradigm for users now.

G: Comments added to the talk page could link to specific article text.

C: We could rethink the talk page so that it becomes an annotation layer of the article.

G: Sometimes you want to see all the comments together though. For readability you want the width of the text to be narrow. But we also want to take advantage of bigger screens these days. We could have a second sidebar with the quality toolbox, or "what are people saying about this article" or "rate this article" or even talk page sections.

P: Here's Diigo working. It's a Firefox plugin, you can see that some people have written notes about the main page on Wikipedia. Here's Cohere, it gives a visual map of the links, you add them manually.

A: If lots of users click a link in an article we could generate an image in some automated way.

... (more to come)