Funder Data Platform
On the 1st December we welcomed around 170 participants in an online workshop to unpick the evidence around experimenting with the funding decision process. The event was co-hosted with the Swiss National Science Foundation (SNSF) and the European Molecular Biology Organization (EMBO).
Together, we explored two of our latest working papers – The experimental research funder’s handbook and Why draw lots? – as well as four case studies of funders experimenting with partial randomisation.
Read the speakers’ slides from the event and below we share some of the discussion from the day.
June 2022 update: An updated version of The experimental research funder’s handbook is now available.
Funders interested in experimenting with grant allocation don’t have to start from scratch. In part 1 of The experimental research funder’s handbook, we describe and signpost tools that can be used to build robust funding experiments. For example, helping organisations know how to gather information, understand the problem, and design a testable intervention.
To make it easier to turn theory into practice, the handbook walks through a series of case studies where these tools and methods have been used to investigate common problems faced by funders – such as devising experiments to explore issues with attracting applicants, selecting reviewers and assessing proposals.
An increasing number of funders are experimenting with new ways of allocating grants. Sharing what they did paves the way for other funders to consider whether experimenting is right for them and what might be the best way to design and evaluate interventions.
Attendees heard from four funders who have experimented with partial randomisation – a process that complements peer review for allocating research funding by applying a randomisation process at particular points, in order to select among recommended proposals.
The design of the randomisation process varied across each case study and raised a range of in-depth questions on how to run experiments in practice, such as whether it’s better to draw lots using a manual drum or software? And should successful applicants be told whether they’re selected by a review panel or draw?
Hearing from those who have already experimented with their funding models helped attendees understand the reality of doing it themselves. For example, it was a surprise to learn that partial randomisation does not necessarily save time for applicants or panel reviewers in practice, compared with more conventional funding approaches.
Speakers shared that surveys of applicants and evaluations have so far found good levels of acceptance of the use of partial randomisation in their schemes. Two of the funders have expanded their use of partial randomisation into more schemes.
You can read the full case studies in part 2 of The experimental research funder’s handbook.
To date, there has been limited research into how people involved in the scientific funding process feel about partial randomisation. Our report, Why draw lots?, is the first time motivations have been compared across funders who currently use – or plan to use – partial randomisation.
Attendees felt reassured that fairness – both within the funding system and from the perspective of applicants – is the strongest theme to emerge from our study. However, it was acknowledged that there’s more nuance to tease out around levels of acceptance and the emotions around funding decisions. For example, do attitudes towards randomly allocated funding differ if an applicant has a good or poor relationship with a funder? And is acceptance of the process higher among applicants who also view peer review as a type of lottery?
While arguments for using partial randomisation in research funding are increasingly well known, and trials considered successful, several unanswered questions and challenges remain around its widespread use:
These questions and more are fruitful areas for further research.
After a fascinating discussion, attendees and speakers reflected on the need to come together to explore the pragmatic realities of experimenting with grant allocation. Working with RoRI and our community of partners helps to boost capacity and resources to tackle shared issues across the research ecosystem.
Together with our partners, we’re co-designing new experiments in research funding to get underway in 2022. Get in touch if you’d like to work with us to explore how you can use experiments, access support to design an experiment, or learn from experiments already underway.
Read more in our summary and a news item in the Times Higher Education.