Democratising funding evaluation can slash the time it takes to deliver a final decision by over 50%

A new RoRI study tests a method for allocating research funding which challenges applicants themselves to decide who gets awarded funds. This study of “distributed peer review” shows that it can be faster, fairer and win support from researchers.

Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations

Read here

Applications to research funding schemes are usually assessed by a panel of external expert reviewers. However, it can be challenging to find sufficient relevant reviewers for a given funding call, and the review process can be time-consuming and costly.

An alternative mechanism is distributed peer review (DPR), where applicants for a funding round also act as reviewers for the other applications. 

People may be familiar with this kind of judging from the long-running UK TV show ‘Come Dine With Me’, where a group of strangers take turns to host a dinner party and rate each other to win a cash prize.

This approach was originally developed to allocate research time on astronomical telescopes in a fairer and more efficient way, but is increasingly being trialled more widely across the research funding ecosystem.

Putting distributed peer review to the test

In the new study, researchers from RoRI worked with the Volkswagen Foundation to compare distributed peer review with conventional panel review for their  ‘Open Up – New Research Spaces for the Humanities and Cultural Studies’ funding call. 

Around 140 proposals were received (an increase on previous years), suggesting applicants were not deterred by the addition of distributed peer review.  

Overall, using DPR cut the average amount of time any individual spent reviewing applications by around 60%, and reduced the average combined time any individual spent reviewing and attending the panel meeting by approximately 87%.

Each proposal also received more attention, with at least 3 times more time spent considering each proposal under DPR compared with the typical time taken for conventional peer review. New applicant-reviewers were able to implement the scoring criteria with the same consistency as experienced panel reviewers, and 8 out of 10 of the proposals selected for funding by DPR made it to the final panel review stage of the process.

83% of funded and 60% of unfunded applicants were positive about participating in future funding calls using this review mechanism, and submissions to the funding call increased by 18% between the first and second trial of DPR, indicating its acceptability.

The study goes beyond previous trials simply testing the feasibility of DPR to running both conventional and DPR review in parallel, allowing a direct comparison of the two methods on the same proposals – something that has not been done before. This is also the first use of DPR within a social sciences setting, showing that this method can work for an interdisciplinary and multidisciplinary scheme. 

The researchers write:

Our research shows that DPR can potentially shorten the time taken for grant applications to be reviewed and funding awarded whilst at the same time providing useful and constructive feedback to those involved, whether or not their applications are successful. 

Overall, the trial underscores the potential of DPR as an innovation which builds on the established and fundamental principle of peer review, but deploys it in a more ‘democratic’ rather than ‘gatekeeping’ mode.

Further adoption of distributed peer review

The use of distributed peer review is set to be scaled up by UK Research and Innovation (UKRI), following a trial that showed the process more than halved review time for grant applications and boosted participation among reviewers. 

Research funders elsewhere in the world have also experimented with DPR, including in the US National Science Foundation (NSF) and National Institute of Food and Agriculture (NIFA), and the Dutch Research Council (NWO). 

Much of this work is summarised in a new RoRI guidebook, Applicants as Reviewers: A Guide to Distributed Peer Review. This is a straightforward guide explaining the benefits and risks of using distributed peer review to speed up and improve research funding processes, along with practical tips for implementation. 

Distributed peer review is just one part of RoRI’s AFIRE project (Accelerator For Innovation & Research Funding Experimentation), aimed at supporting the design, implementation and synthesis of experiments with funding across research institutions. 

Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations

Read here