Topic on Talk:Article feedback

Gaming the system, is there a way to have both useful ratings and conversion of readers to editors?

8
Wittylama (talkcontribs)

According to the Article Feedback Dashboard stats on the 11th of October, the article with the highest average rating in the entire encyclopedia was w:Rangers F.C. (with a 5.0 rating) whilst the lowest rated article is - surprise surprise - their crosstown rival w:Celtic F.C. (with a 1.1 rating). This is possibly the most obvious "gaming the system" of the AFT system there is but even the other articles at the "top" and "bottom" show evident gaming. Other "top" include a series of South Asia related articles including w:List of Nalanda College Colombo alumni, w:Nalanda Maha Vidyalaya Colombo, w:2011 Champions League Twenty20, w:2011 FIBA Asia Championship. Other "bottom" ranked articles include w:Justin Bieber, w:Barack Obama, and w:United States.

The Article Feedback tool seems primarily designed to try to convert readers into editors. However, in the mean time, the feedback that it is generating is *actively* unhelpful. I'm all for converting more readers to becoming editors but the tool as it currently is does not seem to be generating useful ratings information. Is there a way to have both the cake and eat it too? Wittylama 04:45, 11 October 2011 (UTC)

Eloquence (talkcontribs)

You'll always have this kind of rating bias in the data -- one common way to deal with it is to apply smarter algorithms, e.g. to predict individual rater reliability based on past rating behavior.

We've now also started digging more seriously into alternative models for feedback systems that we'll want to experiment with. We've established a contract with Fabrice Florin (of newstrust.net) as product development consultant for this purpose, and you can see some wireframes and ideas in his October 10 presentation. I've encouraged Fabrice to start posting here as well.

Wittylama (talkcontribs)

One of my questions earlier on at this talkpage was about the RFP for the "extended" article feedback tool: Talk:Article_feedback#Extended_aticle_feedback_RFP_7581. Does this mean Fabrice is the person who has been given the project?

With regards to the presentation, linked above, I really like the idea of a new "reviews" tab in the skin especially if it will enable all the functionality identified in the "extended AFT" wireframes and in the presentation :-) I may have overlooked it, but is there any intention of integrating the AFT with the existing Article Rating system in use on talkpages (the FA, GA, B, C, Start, Stub system), or will it be kept separate? I think it would be a shame to have two completely independent ratings systems as that would merely perpetuate what we currently have now - one rating system for readers (that is scorned by users) and one rating system for users (that is unknown to readers). Wittylama 05:19, 13 October 2011 (UTC)

Jorm (WMF) (talkcontribs)

This is the point in which you make such suggestions, I believe. Fabrice is (afaik) managing the project, with a lot of input from us at the WMF and the community. We really want this to be transparent in design, with input from the community, so opening a thread with your thoughts is probably good.

Personally, I'm not a fan so much of a whole "tab" devoted to the reviews. I'd prefer to have the reviews more tightly coupled with the tool itself. Currently, the tool has two modes: "Submit Rating" and "View Ratings"; I could see a third option for "Reviews" or "Suggestions for Improvement" (which is why I argued for a more 'tab' focused format in the current design. This did not manifest, however.).

I do not believe that there is any intention to integration with the current system of article assessment. Article assessment is done by people who are devoted to it or are (supposedly) subject-matter experts. This tool is aimed at readers.

What I'd personally like to see is something that could help to create a "work list". "Needs more photos." "Needs better copy." Etc. Ways in which the reader community can contribute easily. We know many readers have suggestions for improvement but honestly feel like they do not have the right to edit the page. So this can help to bridge that gap.

I think we need to make it clear that there is a difference between the article's "Assessment" and it's "Reader Opinion." Though I'm not entirely convinced (at this point) that there is any "correct" way to do this.

I totally encourage you to open new threads with suggestions for Fabrice. I don't know if you've met him; he's a great guy and totally open to good ideas and constructive discussion.

Wittylama (talkcontribs)

I think you're really onto something with the "work list" as this is (potentially) valuable input that meets both the points of engaging the reader to contribute (a bit, hopefully more) and also providing actionable information for the existing community. You're right that "review" tab isn't necessarily part of the tool and it would be difficult to implement policy-wise because it would introduce a whole new workflow for the editing community. Nevertheless, the implication of having the AFT is that reviews will be made available somewhere and there are disadvantages to having them "dumped" on the talkpage just as there are disadvantages to creating a new "review" space.

I terms of "suggestions to Fabrice"... I've got three things, but I'll put them in a new thread, as you suggest :-)

WhatamIdoing (talkcontribs)

I think it's worth remembering that the Dashboard is designed to identify skewed examples. What you see on the Dashboard is intentionally not representative of normal ratings.

If you want to see more typical uses of the rating system, look at pages like en:Asperger syndrome, en:DNA, or en:Leonardo da Vinci.

Fabrice Florin (talkcontribs)

Thank you all for your thoughtful suggestions about the Article Feedback Tool. It's a real pleasure to meet you, and I am sorry that I couldn't join this conversation sooner. (I just started working for Wikimedia recently as product development consultant on this project, and am still getting oriented.)

Wittylama, I appreciate your concerns about the potential for gaming ratings on Wikipedia. As Eloquence points out, there is always a risk of bias in user ratings like these, but we are exploring a number of remedies to address this issue.

Our current direction is to de-emphasize the ratings in the next version of the AFT, and to invite readers instead to offer specific suggestions for improvement (so their feedback can be more constructive and useful to editors). You can see some examples of that new direction in this new slide show from Oct. 11th, which we updated based on recommendations from the Wikimedia team. To that end, we're looking at services like GetSatisfaction.com for inspiration.

I also really like Jorm's "work list" idea and will aim to integrate that concept in our next round of designs. In coming days and weeks, we will post more wireframes and prototypes -- and seek more feedback from the Wikipedia community, including everyone on this thread. And I will respond next to Wittylama's questions in the newer thread you just posted. Feel free to ping me if you have any more recommendations: we thrive on feedback!

Von Restorff (talkcontribs)

If you thrive on feedback, please do respond to it, even if you disagree with it.

Reply to "Gaming the system, is there a way to have both useful ratings and conversion of readers to editors?"