What do research funders think about experimenting with grant allocation?

On the 1st December we welcomed around 170 participants in an online workshop to unpick the evidence around experimenting with the funding decision process. The event was co-hosted with the Swiss National Science Foundation (SNSF) and the European Molecular Biology Organization (EMBO).  

Together, we explored two of our latest working papers – The experimental research funder’s handbook and Why draw lots? – as well as four case studies of funders experimenting with partial randomisation.   

Read the speakers’ slides from the event and below we share some of the discussion from the day. 

June 2022 update: An updated version of The experimental research funder’s handbook is now available.  

Knowing where to start – using the right tools and resources 

Funders interested in experimenting with grant allocation don’t have to start from scratch. In part 1 of The experimental research funder’s handbook, we describe and signpost tools that can be used to build robust funding experiments. For example, helping organisations know how to gather information, understand the problem, and design a testable intervention.

To make it easier to turn theory into practice, the handbook walks through a series of case studies where these tools and methods have been used to investigate common problems faced by funders – such as devising experiments to explore issues with attracting applicants, selecting reviewers and assessing proposals. 

Lessons learned from case studies  

An increasing number of funders are experimenting with new ways of allocating grants. Sharing what they did paves the way for other funders to consider whether experimenting is right for them and what might be the best way to design and evaluate interventions. 

Attendees heard from four funders who have experimented with partial randomisation – a process that complements peer review for allocating research funding by applying a randomisation process at particular points, in order to select among recommended proposals.

The design of the randomisation process varied across each case study and raised a range of in-depth questions on how to run experiments in practice, such as whether it’s better to draw lots using a manual drum or software? And should successful applicants be told whether they’re selected by a review panel or draw? 

Hearing from those who have already experimented with their funding models helped attendees understand the reality of doing it themselves. For example, it was a surprise to learn that partial randomisation does not necessarily save time for applicants or panel reviewers in practice, compared with more conventional funding approaches. 

Speakers shared that surveys of applicants and evaluations have so far found good levels of acceptance of the use of partial randomisation in their schemes. Two of the funders have expanded their use of partial randomisation into more schemes.  

You can read the full case studies in part 2 of The experimental research funder’s handbook. 

More research is needed into randomisation approaches for scientific funding

To date, there has been limited research into how people involved in the scientific funding process feel about partial randomisation. Our report, Why draw lots?, is the first time motivations have been compared across funders who currently use – or plan to use – partial randomisation. 

Attendees felt reassured that fairness – both within the funding system and from the perspective of applicants – is the strongest theme to emerge from our study. However, it was acknowledged that there’s more nuance to tease out around levels of acceptance and the emotions around funding decisions. For example, do attitudes towards randomly allocated funding differ if an applicant has a good or poor relationship with a funder? And is acceptance of the process higher among applicants who also view peer review as a type of lottery? 

Known unknowns: what we still need to find out about randomisation in research funding

While arguments for using partial randomisation in research funding are increasingly well known, and trials considered successful, several unanswered questions and challenges remain around its widespread use:

• How do funders decide when it’s right to expand the use of partial randomisation beyond small-scale trials? Lengthy funding cycles and slow-moving processes mean that the benefits of new approaches can only be measured in the long-term.  

• How do we identify schemes and disciplines best suited for partial randomisation? Interdisciplinary fields, fields particularly vulnerable to bias or requiring high innovation and where it’s hardest to agree on criteria may benefit more from randomised funding approaches.

• What are the unintended consequences or knock-on effects? For example, how might partial randomisation affect research culture around peer review and research assessment? And will it affect the way projects and applications are developed? 

These questions and more are fruitful areas for further research. 

Opportunities to get involved

After a fascinating discussion, attendees and speakers reflected on the need to come together to explore the pragmatic realities of experimenting with grant allocation. Working with RoRI and our community of partners helps to boost capacity and resources to tackle shared issues across the research ecosystem.

Together with our partners, we’re co-designing new experiments in research funding to get underway in 2022. Get in touch if you’d like to work with us to explore how you can use experiments, access support to design an experiment, or learn from experiments already underway.

Read more in our summary and a news item in the Times Higher Education.