RoRI Randomisation project
How can research funders be more experimental? Are lottery-style funding mechanisms a good idea?
Summary
Billions of euros are spent on research every year, but we don't know if this is allocated in the most effective or fair way. Worldwide, there is growing interest in using experimental approaches for funding research, such as 'lottery-type' mechanisms, but we don't know how well these work or which situations they might suit best.

Our RANDOMISATION project worked with RoRI partners running experiments in grant allocation. It collated lessons learned and practical information to help other funders decide whether experimenting is right for them and how to best design and evaluate interventions.

About the research
We used a variety of methods including reviewing the literature and collecting information through workshops and interviews with funders actively running experiments in grant allocation.

The project had three aspects:

1) Exploring how experimental approaches can be used to address a number of common challenges that funders face, and identifying examples of how experiments have been used to tackle them. We explore the steps, tools and approaches that funders can use when setting up experiments.

2) A small-scale qualitative study exploring the motivations, drivers and constraints around partial randomisation in research funding. This captures the views of senior managers and practitioners in funding organisations to better understand the motivations behind experiments, and how people feel about it.

3) Supporting a number of RoRI partners with the design and evaluation of experiments, and helped others to build the internal strategic case for trials. For example, we ran a workshop with the Research Council of Norway to provide international context to help support their plans for an initial partial randomisation trial.
Who was involved?
The following RoRI partners and organisations took part in this project:

Australian Research Council
Alfred P. Sloan Foundation
Austrian Science Fund
Chan Zuckerberg Initiative
European Molecular Biology Organization
Michael Smith Foundation for Health Research
National Institute for Health Research
Innovation Growth Lab at Nesta (non-RoRI partner)
Netherlands Organisation for Scientific Research
Novo Nordisk Fonden
Research Council Norway
Swiss National Science Foundation
UK Research and Innovation
Volkswagen Foundation
Wellcome Trust

Download the research
This work is made up of two reports:

1) The experimental research funder's handbook

Part 1 explores how funders could become experimental. It introduces practical tools and methods for diagnosing issues and designing experiments - including examples of how to use them. Part 2 takes a closer look at partial randomisation, drawing on experiments carried out by four funders. It lists the benefits, pros and cons and a checklist of essential steps.

2) Why draw lots? Funder motivations for using partial randomisation to allocate research grants

Results from a small-scale qualitative study exploring motivations and restraining factors around experimental approaches to grant allocation. Based on interviews with practitioners and leaders from six funding organisations either planning or implementing partial randomisation.

Some of our early work can be found in our working paper Experiments with randomisation in research funding: scoping and workshop report.
Find out more
In December 2021, we hosted an event sharing this research.

See again the speakers' slides, read discussion highlights and a detailed summary.

Introduction to partial randomisation
While there are many new approaches to grant allocation, this project took a closer look at partial randomisation in particular - an approach that's increasingly being trialled.
What is partial randomisation?
What is partial randomisation?
Partial randomisation is a mechanism that complements peer review for allocating research funding. It's also called focal or targeted randomisation, or a modified lottery.

A randomisation process is only applied to a subset of applications - hence, the term partial, focal or targeted.

It relies on peer reviewers' expertise to first recommend applications for funding - those that meet the quality and criteria set by the grant. Randomisation is then applied to select among the recommended applications.

There are slight variations of partial randomisation. For example, randomisation can be applied after one round of peer review where all recommended applications are entered into a draw. Or the review panel defines a threshold, above which applications are funded and those below enter a randomised selection process.

The tools used to deliver randomisation also vary - from manual lottery drums to software.
What are the benefits of partial randomisation?
The benefits of partial randomisation
Using peer review processes in grant allocation can have its limitations. Partial randomisation offers a way to overcome some of these limitations while retaining expert reviewers' judgment as the main mechanism to assess an application's quality.
Suitable for highly competitive grants
Reduces reviewer fatigue
Avoids the need to distinguish between similar, 'grey area', applications
Removes pressure for reviewers to be an expert in multiple fields
Not biased against high-risk research
What are the pros and cons?
The pros and cons
Building on the lessons learned by RoRI partners, below lists the most common arguements for and against introducing partial randomisation in grant allocation.
Can eliminate bias and increase diversity
Goes against merit-based decision making
Can foster innovative and creative research which is sometimes considered high-risk
Could reduce the quality of applications
Reduces the burden on reviewers and potentially applicants
Risk of undermining trust in, and credibility of, the research system
Increases transparency of the selection process
Risk of stigma and reputational damage for researchers, schemes and funders
The selection process can be faster and cheaper
Can transfer decision-making away from researchers and onto administrators
Less stigma for rejected applicants

Learning from others: case studies
We present four case studies of recent experiments with partial randomisation. Explore why funders chose to use partial randomisation, details of the procedures they followed, evaluation results, and reactions from applicants and reviewers.

For the full case studies, see Part 2 of The experimental research funder's handbook.
Volkswagen Foundation
Austrian Science Fund
Swiss National Science Foundation
Health Research Council of New Zealand
The Volkswagen Foundation introduced partial randomisation as a trial in its Experiment! – In Search of Bold Research Ideas funding initiative in 2017. The trial ended in 2020 coinciding with the Foundation's development of a new funding strategy.

The Experiment! scheme supported high-risk exploratory research within the life sciences, natural sciences, and engineering - funding creative and unconventional research projects that would not have easily passed conventional peer review.

From its beginning, the scheme followed an unconventional selection process to account for applications lacking reliable preliminary work. From the 2017 round, partial randomisation was also introduced and involved the use of a physical lottery drum.

Use of partial randomisation is considered a success. The scientific community was open to a change, and preliminary evaluation findings do not show major drawbacks.
The Austrian Science Fund uses partial randomisation in its 1000 Ideas Programme - an ongoing pilot grant scheme that started in 2019.

The 1000 Ideas Programme provides seed funding for radically new and bold research ideas in all disciplines that have the potential to transform established scientific knowledge, but has a hard time obtaining funding from traditional schemes.

Partial randomisation has been used since the start of the programme - to reduce the risk of bias in decision making. The partial randomisation procedure uses 'R' software (package 'dplyr') and takes place after applications are evaluated and scored by the Austrian Science Fund Board and a jury.

Initial feedback about the selection process from applicants, particularly from early career researchers, has been generally positive. A formal evaluation of the 1000 Ideas programme will be carried out when a larger sample of applications is available.
The Swiss National Science Foundation (SNSF) trialled partial randomisation in its Postdoc.Mobility scheme between 2018 and 2019.

The Postdoc.Mobility scheme is a fellowship programme for postdoctoral researchers who want to deepen their scientific knowledge and increase scientific independence through a research stay abroad.

The experiment used partial randomisation as an optional, final step to fund or reject proposals of similar quality close to the funding cut-off, only when the panel could not reach a decision. A process of manually drawing lots was followed.

An internal evaluation found applicants generally accepted the procedure. Further evaluation will be done including looking at the impact on career development comparing applicants who received funding with those who were rejected by partial randomisation.

Building on what was learned by this trial, SNSF will harmonise its evaluation procedures across all of its funding schemes - including the optional use of partial randomisation. SNSF is the first national funder that allows partial randomisation to be applied in all of its schemes. A digital tool for partial randomisation might be used in the future after having ensured that no bias or manipulation can occur.
The Health Research Council of New Zealand (HRC) was the first public funder to experiment with partial randomisation - trailing it in the Explorer Grants scheme since its first call in 2013.

Explorer Grants fund potentially transformative ideas at an early development stage, in any health research discipline. The scheme is designed to support applications not fundable through other HRC schemes because of their unpredictability and lack of supporting data.

Unlike other experiments, partial randomisation is used to select applications to fund from the entire pool of fundable proposals - after the Assessment Committee confirms main criteria are met. All fundable applications are assigned a random number using the RAND function of Microsoft Excel and they are funded in the order of smallest to largest random number until the available budget is exhausted.

HRC's use of partial randomisation is continually evaluated. One formal evaluation found a general level of acceptance, that applicants spend the same length of time preparing applications, and that HRC staff had a reduced administrative burden.

HRC has expanded the use of partial randomisation into two other schemes - although the availability of funding has not made its use necessary yet.

What's next?
We're continuing to collate and synthesise lessons learned from ongoing experiments. We'll add these and more resources to The experimental research funder's handbook. We're interested in hearing what other experiments might be happening and resources funders are using. Let us know if you've got examples we can add to the handbook.
With partners, we're co-designing new experiments to get underway in 2022. Get in touch if you'd like to work with us to explore how you can use experiments, receive support to design a new experiment, or learn from experiments already underway.
Made on
Tilda