User:BDavis (WMF)/Projects/Review tool

Review tool is a placeholder name for a system for managing the process of having a group of users evaluate grant/scholarship/whatever proposals and collecting the evaluations to use in making final decisions.

Problem
The Wikimania Scholarships app is a one-off solution to this problem. IEG would like something similar. Internal discussions at the WMF indicate that if a more general self-serve solution could be found there would be additional programs interested in using it.

Simplifying assumptions

 * For an MVP, there is no need to collect information from applicants
 * For an MVP, administrators can be responsible for populating list of proposals
 * For an MVP, there will be no localization of string content entered by administrators (field labels, etc)
 * For an MVP, getting the IEG workflow to function with no UI to create additional campaigns or configure the proposals and reviews is acceptable

User stories

 * As an administrator of the grant review system
 * I want to create new grant campaigns (defining reviewers, review criteria, data to be reviewed, and data grouping)
 * So that reviewers can provide feedback on the applications in the campaign.


 * As a grant administrator or reviewer
 * I want to mark grant applications as eligible or ineligible
 * So that ineligible grant applications can be excluded from further review.


 * As a grant reviewer
 * I want to score a grant application on multiple dimensions (defined by the admin)
 * So that the quality of the grant application can be compared to others in the same campaign.


 * As a grant provider or administrator
 * I want to view reports on the aggregate reviewer scores for grant applications in a campaign
 * So I can share results with others outside of the review system and select applications to fund.

Hypothetical workflow

 * Users can invite new users to the system
 * Users can authenticate via OAuth
 * Users can create a password to authenticate in the event of OAuth failure


 * Review tool admin creates a new campaign and assigns 1 or more users as campaign administrators


 * Campaign administrator fills in campaign details
 * Define what a "proposal" looks like for this campaign by describing fields that can be populated when a proposal is entered
 * Define what a "review" looks like for this campaign by describing fields that can be populated when a review is entered
 * Define what the review queue looks like for this campaign (which fields from the proposals are shown as columns and what order)
 * Define ... other things about the campaign
 * Import/create proposals matching definition for this campaign
 * Assign 1 or more users as reviewers for this campaign


 * Reviewers see list of campaigns they can review proposals in
 * A paginated, sortable list of proposals belonging to a campaign is available to reviewers
 * A reviewer selects a proposal and sees a screen giving proposal details and a review form
 * A reviewer submits review and can see their feedback along with anonymized feedback from other reviewers


 * A campaign administrator can see non-anonymized feedback from all reviewers
 * A campaign administrator can see summary report(s) of all reviews
 * A campaign administrator can export all reviews as CSV for import into another tool
 * A campaign administrator can close the campaign which stops new reviews but allows browsing reports and individual proposals and reviews.
 * A campaign administrator can publish the campaign which closes the campaign and makes anonymized review feedback available to non-authenticated users.
 * A campaign administrator can archive the campaign which closes the campaign and removes rights for anyone but administrators to view data.
 * A campaign administrator can delete a campaign which removes data from the backing datastore in a non recoverable manner.