Article feedback/Public Policy Pilot

This page lists the Functional Requirements for the Public Policy Pilot of the Article Assessment feature. This is a work in progress.

Feedback from the Public Policy Initiative has been and will continue to be incorporated into this feature. The Public Policy Initiative will decide whether to use the data as part of their project evaluation.

Objectives

 * Encourage user participation by giving readers the ability to assess and article
 * Give readers and editors a glanceable average of (user generated) ratings

Timeline
We aim to have this feature in production by September 15, 2010.

Use Cases
(To be completed)

Feature Requirements
General requirements
 * This feature will exist only on a subset of pages designated by the Public Policy Initiative
 * This feature will be available to all users (not just logged in)

Reader Feedback: Input and Display
 * There will be the following 4 categories, each with a five-star rating:
 * Well-sourced
 * Complete
 * Neutral
 * Readable


 * There will be tooltips to describe each category of rating (Amy to provide descriptions)
 * There will be a link to provide feedback on the feature. This link will go to a survey (Proposed questions  here (Survey page here)


 * For a given user and article, there are the following states for review:
 * Haven't reviewed at all
 * Have reviewed current revision, let user change review
 * Haven't reviewed current revision, but reviewed previous revision, let user re-review, give message that their review is stale


 * There will be two areas (boxes) upon page load:
 * Rate this article box: for reader to provide ratings:
 * Averages will not be reflected in stars
 * Once user submits ratings, page will reload and reflect action just taken
 * Number of ratings (below) increments
 * Average ratings box: displays average of ratings + total number or ratings per dimension in graphical form
 * Averages calculated by taking average of last 10 ratings. If there are fewer than 5 ratings, we will display "Not enough ratings to provide average" message.
 * There will be links to view the average ratings in graphical form.


 * Re-rating an article (stale ratings)
 * Once a user has rated an article, they will be prevented from rating it again unless the stale requirement has been met
 * Stale requirement: if there has been at least 5 reviews from other users since the last review, we will inform the user that their review may be stale (messaging needs to be soft and cannot imply that the article is stale). The user may then re-rate the article.


 * Updating Article Averages (stale averages)

Analytics
 * Ideally, we would have page views per article so that we know the ratings rate = (number of ratings) / (total views of page)
 * We also need some measurement of ratings completion = (users that complete the rating process) / (users that start the rating process)
 * Ways of analyzing ratings distributions in aggregate (vs. one article at a time).

Known Issues
 * Re-rating current revisions should be allowed. This is a bug in the existing software.

Deleted Requirements
 * THere will be 1 "overall" rating with a five category rating
 * There will be an open comment field. What happens with the information entered in the comments field is TBD. Options are:
 * The comment goes into some database, but not published anywhere (I think this is where we are leaning)
 * The comments go a special page which is linked from somewhere either on the article page or rating control
 * We'll likely see spam quickly. Restricting addition of URLs would help. Note that WP also has a fair share of aggressive trolls/vandals who may post inappropriate or personal information. If we don't want to implement a basic deletion mechanism, we will probably need to restrict views.

API
The following API calls will be used with

getCumulativeResults
invoked by  where pageid is the page ID and revid is the revision ID of the desired page

This will return a cached result in an object nested like so  Where 1-4 are the indicies for the dimensions of the review, avg is the average for that dimension, and total is the total number of reviews

setUserVals
invoked by where reviewcookie is the unique identifier for this user in the review system, authcookie is a cookie to match against the server to avoid CXRF, pageid is the page ID, revid is the revision ID of the desired page, and djsonRevObj is a review object in JSON of the following: where val are the values the user has assigned

Database
The data stored will be in the following tables:

article_assessment
Will hold 1 entry per user per revision, namely: Where m1...mn are the dimensions we're measuring (completeness, npov, etc)

article_assessment_pages
Will hold 1 entry per page revision per dimension we're measuring, namely: Where dimension refers to the dimension that's being measured (1 = completeness, etc)

Assumptions

 * previous user ratings will be stored in a cookie on the user's browser. All that needs to be stored is [pgid, revid, dimension, rating] but only for the last revision that has been given
 * the page's pageid and current revid need to be visible to JS as variables

Limitations

 * Any user who uses a different browser or clears their cookies will be seen by the system as having a different browser
 * After the user hits the maximum number of reviewable pages, they won't be able to see their ratings for the first reviewed page
 * We need to limit the MWHooks to 1 call so that code only gets injected when it is a page that has AA enabled for it, and this comparison only needs to be made once

Open Issues

 * Cache invalidating - On every new rating, the cached version of this page will have to be invalidated.
 * We mark pages as invalid after an edit, short-term solution is to do this
 * Long-term solution is to get the ratings info from an API call that has cache-able resources


 * Only to appear on select pages
 * Short-term solution is to have a configurable list of page titles or IDs that this extension will be visible on, and optimize to exit out if the page is not among these
 * Long-term solution not necessary as this will be on all pages long-term