AFIRE (Accelerator For Innovation & Research Funding Experimentation) is RoRI’s programme for the design, implementation and synthesis of experiments with research and innovation funding. We partner with the Innovation Growth Lab (IGL) in order to:

• Host forum events to showcase experimental studies conducted in funding organisations
• Organise ‘Sprint’ events to develop institutional capacity to design and carry out experiments
• Design experiments ourselves and carry them out across multiple funding organisations, simultaneously where feasible

Within AFIRE, we are currently catalysing, conducting or evaluating experiments on distributed peer review, partial randomisation, AI-assisted reviewer allocation, and desk rejection.

“Good funder experiments take rigour and imagination. It’s exciting to be part of a project which is able to draw on both, from across the RoRI network of expertise. Together we’ll be able to support more experiments and better use of experimental evidence to improve research funding.” (Tom Stafford, project lead)

Project team

• Tom Stafford (RoRI) – Project lead
• Peter Kolarz (RoRI, UCL) – Project co-lead
• Albert Bravo-Biosca (IGL, Nesta)
• Theodore Hodapp (Gordon and Betty Moore Foundation) – Co-Chair
• Stian Westlake (UKRI) – Co-Chair
• Josie Coburn (RoRI, UCL)
• Cindy Lopez-Bento (KU Leuven)
• Ines Rombach (RoRI, University of Sheffield)
• Anna Butters (RoRI, University of Sheffield)
• Melanie Benson Marshall (RoRI, University of Sheffield)
• James Wilsdon (RoRI, UCL)
• Stephen Curry (RoRI)
• James Phipps (IGL, Nesta)

Partner organisations

Project partners include: 

• The Gordon and Betty Moore Foundation
• Luxembourg National Research Fund (FNR)
• Innovation Growth Lab (IGL)
• Wellcome Trust
• Volkswagen Foundation
• UK Research and Innovation (UKRI)

The project in detail

Interest in experiments is growing, but different research funders are at different stages of readiness as reported by a RoRI-SNSF-EMBO-IGL workshop in December 2021 (internal report ‘RoRI Funder Lab: improving research funding processes through experimentation’).  AFIRE provides a broad platform accessible to funders at all levels of engagement, combined with structured support to work with funders to increase the speed and ambition of their engagement with experiments. Our aims are to enhance:

• Awareness of experimental possibilities and findings in the funding ecosystem.
• Motivation to experiment, and to evaluate the effectiveness of interventions
• Capacity to experiment, encompassing both technical expertise around issues such as data collection and analysis, as well as an understanding of the frontiers of research on research and innovation funding so interventions can be targeted to the most pressing needs.
• Scale and reach of experiments. The Accelerator will support getting experiments up and running, with greater ambition in running experiments at larger scale, at greater levels of robustness and in ways that support sharing and dissemination of post-experiment evaluation.
• The culture of research and innovation funding, across the whole sector, so it is more experimental, creative and evidence-informed.

AFIRE Forum

The AFIRE Forum is an arena for peer exchange and learning about experiments at ideation, design stage or in progress. The first event in this series was held in May 2024. Invite only, and conducted under the Chatham House Rule, these informal discussions provide a safe space in which the real work of experimental approaches can be discussed in detail.

AFIRE Sprints

Sprints are on-demand small-group mini-workshop series tackling a particular issue, question, or experimental stage. Sprints involve more intensive and targeted activity with the intention to achieve substantive progress over a short period of time. They bring together a group of partners who are developing experiments within the same topic area or who are at a similar stage in the experimental process. Each sprint consists of a series of sessions facilitated by our team, with participants expected to work independently and make progress on their projects in-between sessions.

The starting point varies for each sprint, and the approach, format and frequency of meetings is adjusted depending on the identified objective, participants’ needs and their availability. Some sprints may bring together multiple partners developing experiments around a similar challenge (e.g., biases) while others may focus instead on helping partners make progress on a particular experimentation stage (e.g., developing a logic model and seeking user feedback).

While the intention is for sprints to bring together multiple partners in order to benefit from peer learning opportunities, there is the option of having sprints that only include one active partner if it is difficult to align timetables and priorities across multiple partners. 

Experiments

Distributed peer review

A major strand of experimental work in AFIRE has been on distributed peer review (DPR). This intervention involved applicants to a funding scheme also acting as reviewers on the same scheme. RoRI researchers have helped conduct experiments on DPR at the Volkswagen Foundation over the past year, with additional experimentation now in progress.

Findings from this work have been widely publicised, and plans are taking shape for RoRI to repeat the experiment at other funding organisations, and to produce guidance and indicator suites for other funders to replicate the experiment by themselves

Partial randomisation

As a precursor to AFIRE, RoRI’s earlier work on funding process experimentation focused on partial randomisation. This work resulted in The Experimental research Funder’s Handbook, published in 2023. It showcases experiments with partial randomisation conducted at the Volkswagen Foundation, the Swiss National Science Foundation (SNSF), the Austrian Science Fund (FWF), and the Health Research Council of New Zealand.

We continue to monitor and encourage experimentation with partial randomisation and are working on a core outcomes set for such experiments as a resource for funders interested in designing them.

AI-assisted reviewer selection and allocation

We are engaged in several on-going activities around experimentation on AI-assisted reviewer allocation. A number of Sprints have recently begun around this topic, hopefully leading to experimentation in the future. Through the GRAIL project, we have also collected a range of case studies on various funders’ experiments with AI-assisted reviewer allocation.

We also conducted a live-experiment on the abstract review for the recent Metascience2025 conference, comparing a shadow AI-allocation of abstracts to reviewers with reviewers’ self-reported suitability to each abstract following manual allocation.

The case studies and the Metascience2025 experiment are due to be published in early 2026.

Desk Rejection

Desk Rejection is the practice of using internal staff expertise at the funder to identify funding applications which are unlikely to be successful. Adopting desk rejection is an innovation that offers considerable efficiency savings for both funders and reviewers.

So far, the knowledge and expertise of funders’ grant management/administrative staff (and the potential benefits that this expertise may hold) is a critically under-researched topic. This experiment will contribute to filling that gap.

We are setting up a multi-funder shadow experiment, meaning that the funders’ business-as-usual continues, while an alternative process with no real-world consequence (the ‘shadow’ process) is also conducted in parallel, resulting in direct outcomes comparison between the real and shadow processes. The research question we seek to answer is: How accurately does desk rejection correctly categorise rejection via external review?

Project timeline

The AFIRE was launched in 2024 and will run until the end of 2027.

Outputs

Applicants as reviewers: a guide to distributed peer review

A practical guide for funders exploring or refining Distributed Peer Review, with checklists, setup tips, and lessons from real-world trials

Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations

Paper presenting a mixed-methods evaluation of an experiment with distributed peer review (DPR) carried out by the Volkswagen Foundation

Related publications

• Stafford T, Rombach I, Hind D et al. (2023) Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives Wellcome Open Res 2023, 8:309 (https://doi.org/10.12688/wellcomeopenres.19565.1)
• Bendiscioli, Sandra; Firpo, Teo; Bravo-Biosca, Albert; Czibor, Eszter; Garfinkel, Michele; Stafford, Tom; et al. (2022): The Experimental Research Funder’s Handbook (Revised edition, June 2022, ISBN 978-1-7397102-0-0). Research on Research Institute. Report. https://doi.org/10.6084/m9.figshare.19459328.v2