User:BDavis (WMF)/Projects/Review tool

From mediawiki.org

Review tool is a placeholder name for a system for managing the process of having a group of users evaluate grant/scholarship/whatever proposals and collecting the evaluations to use in making final decisions.

Problem[edit]

The Wikimania Scholarships app is a one-off solution to this problem. IEG would like something similar. Internal discussions at the WMF indicate that if a more general self-serve solution could be found there would be additional programs interested in using it.

Simplifying assumptions[edit]

  • For an MVP, there is no need to collect information from applicants
  • For an MVP, administrators can be responsible for populating list of proposals
  • For an MVP, there will be no localization of string content entered by administrators (field labels, etc)
  • For an MVP, getting the IEG workflow to function with no UI to create additional campaigns or configure the proposals and reviews is acceptable

User stories[edit]

As an administrator of the grant review system
I want to create new grant campaigns (defining reviewers, review criteria, data to be reviewed, and data grouping)
So that reviewers can provide feedback on the applications in the campaign.


As a grant administrator or reviewer
I want to mark grant applications as eligible or ineligible
So that ineligible grant applications can be excluded from further review.


As a grant reviewer
I want to score a grant application on multiple dimensions (defined by the admin)
So that the quality of the grant application can be compared to others in the same campaign.


As a grant provider or administrator
I want to view reports on the aggregate reviewer scores for grant applications in a campaign
So I can share results with others outside of the review system and select applications to fund.

Hypothetical workflow[edit]

  • Users can invite new users to the system
  • Users can authenticate via OAuth
  • Users can create a password to authenticate in the event of OAuth failure
  • Review tool admin creates a new campaign and assigns 1 or more users as campaign administrators
  • Campaign administrator fills in campaign details
    • Define what a "proposal" looks like for this campaign by describing fields that can be populated when a proposal is entered
    • Define what a "review" looks like for this campaign by describing fields that can be populated when a review is entered
    • Define what the review queue looks like for this campaign (which fields from the proposals are shown as columns and what order)
    • Define ... other things about the campaign
    • Import/create proposals matching definition for this campaign
    • Assign 1 or more users as reviewers for this campaign
  • Reviewers see list of campaigns they can review proposals in
  • A paginated, sortable list of proposals belonging to a campaign is available to reviewers
  • A reviewer selects a proposal and sees a screen giving proposal details and a review form
  • A reviewer submits review and can see their feedback along with anonymized feedback from other reviewers
  • A campaign administrator can see non-anonymized feedback from all reviewers
  • A campaign administrator can see summary report(s) of all reviews
  • A campaign administrator can export all reviews as CSV for import into another tool
  • A campaign administrator can close the campaign which stops new reviews but allows browsing reports and individual proposals and reviews.
  • A campaign administrator can publish the campaign which closes the campaign and makes anonymized review feedback available to non-authenticated users.
  • A campaign administrator can archive the campaign which closes the campaign and removes rights for anyone but administrators to view data.
  • A campaign administrator can delete a campaign which removes data from the backing datastore in a non recoverable manner.

Data model[edit]