User:BDavis (WMF)/Projects/Review tool

Review tool is a placeholder name for a system for managing the process of having a group of users evaluate grant/scholarship/whatever proposals and collecting the evaluations to use in making final decisions.

Problem

The Wikimania Scholarships app is a one-off solution to this problem. IEG would like something similar. Internal discussions at the WMF indicate that if a more general self-serve solution could be found there would be additional programs interested in using it.

Simplifying assumptions

• For an MVP, there is no need to collect information from applicants
• For an MVP, administrators can be responsible for populating list of proposals
• For an MVP, there will be no localization of string content entered by administrators (field labels, etc)
• For an MVP, getting the IEG workflow to function with no UI to create additional campaigns or configure the proposals and reviews is acceptable

User stories

As an administrator of the grant review system

I want to create new grant campaigns (defining reviewers, review criteria, data to be reviewed, and data grouping)

So that reviewers can provide feedback on the applications in the campaign.

As a grant administrator or reviewer

I want to mark grant applications as eligible or ineligible

So that ineligible grant applications can be excluded from further review.

As a grant reviewer

I want to score a grant application on multiple dimensions (defined by the admin)

So that the quality of the grant application can be compared to others in the same campaign.

As a grant provider or administrator

I want to view reports on the aggregate reviewer scores for grant applications in a campaign

So I can share results with others outside of the review system and select applications to fund.

Hypothetical workflow

• Users can invite new users to the system
• Users can authenticate via OAuth
• Users can create a password to authenticate in the event of OAuth failure

• Review tool admin creates a new campaign and assigns 1 or more users as campaign administrators

• Campaign administrator fills in campaign details
  • Define what a "proposal" looks like for this campaign by describing fields that can be populated when a proposal is entered
  • Define what a "review" looks like for this campaign by describing fields that can be populated when a review is entered
  • Define what the review queue looks like for this campaign (which fields from the proposals are shown as columns and what order)
  • Define ... other things about the campaign
  • Import/create proposals matching definition for this campaign
  • Assign 1 or more users as reviewers for this campaign

• Reviewers see list of campaigns they can review proposals in
• A paginated, sortable list of proposals belonging to a campaign is available to reviewers
• A reviewer selects a proposal and sees a screen giving proposal details and a review form
• A reviewer submits review and can see their feedback along with anonymized feedback from other reviewers

• A campaign administrator can see non-anonymized feedback from all reviewers
• A campaign administrator can see summary report(s) of all reviews
• A campaign administrator can export all reviews as CSV for import into another tool
• A campaign administrator can close the campaign which stops new reviews but allows browsing reports and individual proposals and reviews.
• A campaign administrator can publish the campaign which closes the campaign and makes anonymized review feedback available to non-authenticated users.
• A campaign administrator can archive the campaign which closes the campaign and removes rights for anyone but administrators to view data.
• A campaign administrator can delete a campaign which removes data from the backing datastore in a non recoverable manner.

Data model