Article feedback/Research

Overview
Here is an update of the Article Feedback Data as of October 4, 2010. It is based on approximately 2,247 ratings submitted from Sep 22 - Oct 4. A running list of articles is maintained here, but please keep in mind the list is subject to change.

Previous versions of this data: Sep 28 Update

Overall Ratings Data
The following table summarizes the aggregate rating data:


 * Completion rates for each category (defined as the number of ratings for the category divided by the total number of ratings) is between 90% and 96%.

Comparing Anon Reviewers to Registered Reviewers
Here are the tables comparing ratings from Anonymous and Registered users:

A few things worth noting:


 * It appears as though registered users are “tougher” in their grading of the articles than are anon users. This is especially notable in the area of “well sourced” (3.8 mean for anon vs. 2.5 mean for registered) and “complete” (3.6 vs. 2.4).  It’s interesting to note that the means for “neutral” are almost identical.


 * The completion rate for reviews continues to be higher for registered users. It’s worth noting that “Neutral” had the lowest completion rate for both registered and anonymous users.


 * The standard deviation of ratings across all categories is lower for registered than for anon. While this appears to suggest that the ratings of registered users are more internally consistent than the ratings of anonymous users, looking at the actual distributions suggests the opposite:



The distribution of the ratings are beginning to show marked differences between Anonymous and Registered Users:
 * Anonymous Users are much more generous with their ratings. 4s and 5s are most common rating across all categories.  These users are far more likely to give 5's than are registered users.  For example, under "Well-Sourced", 45% of the ratings from anonymous users were 5 stars whereas only 10% of registered users rated this category 5 stars.
 * Registered Users show distinct patterns depending on the category:
 * Neutral and Readable: Both these categories show a normal-like distribution around the mean.
 * Well-Sourced and Complete: For these categories, the most common rating is 1, and the ratings fall off in a linear-like fashion from 1 to 5. The perceptions registered users have of these categories appears to be significantly worse than their perceptions of other categories.

10 most frequently rated articles
(Simply sorted by number of submitted "well sourced" ratings.)


 * http://en.wikipedia.org/wiki/United_States_Constitution - 80 ratings -- linked from Wikimedia blog post
 * http://en.wikipedia.org/wiki/Don't_ask,_don't_tell - 61 ratings -- linked from Wikimedia blog post
 * http://en.wikipedia.org/wiki/Capital_punishment - 37 ratings
 * http://en.wikipedia.org/wiki/Terrorism - 35 ratings
 * http://en.wikipedia.org/wiki/United_States_Declaration_of_Independence - 32 ratings
 * http://en.wikipedia.org/wiki/DREAM_Act - 32 ratings
 * http://en.wikipedia.org/wiki/LGBT_rights_in_the_United_States - 30 ratings
 * http://en.wikipedia.org/wiki/5_centimeters - 28 ratings -- third item in public policy category
 * http://en.wikipedia.org/wiki/Pollution - 27 ratings
 * http://en.wikipedia.org/wiki/Abortion - 22 ratings

To Do

 * Breakdown of ratings (particularly num. ratings) by user (username or IP)
 * Top 10 (most rated) article comparison
 * Top 10 (most prolific raters) user comparison
 * Short article (with rating tool visible) Vs. others comparison
 * Short No. 1 (viewable on 1280 X 1024): http://en.wikipedia.org/wiki/Executive_Order_11478
 * Short No. 2: http://en.wikipedia.org/wiki/5_centimeters (stub)
 * Short No. 3: http://en.wikipedia.org/wiki/1984_Cable_Franchise_Policy_and_Communications_Act (stub)
 * Short No. 4: http://en.wikipedia.org/wiki/David_Ray_Hate_Crimes_Prevention_Act (stub)
 * Short No. 5: http://en.wikipedia.org/wiki/Balanced_Budget_Act_of_1997 (stub)
 * Comparison of average ratings to current Wikipedia rating system (FA, GA, etc)
 * Investigate the 87+% 4 metric ratings (forced choice? felt mandatory?  confidence in some over others?)
 * Email questionnaire to users about confidence in the accuracy of their ratings
 * Investigate whether those rating articles have also contributed/edited that article (could be done in the questionnaire)
 * Ask Roan if we can have a cumulative Page View column in our CSV data pull
 * Investigate "neutrality" - changing the word? description? placement?
 * Investigate "completeness"' relation to article length