What are you looking for?

AFIRE

Accelerator For Innovation & Research Funding Experimentation

AFIRE (Accelerator For Innovation & Research Funding Experimentation) is RoRI’s programme for the design, implementation and synthesis of experiments with research and innovation funding. We partner with the Innovation Growth Lab (IGL) in order to:

  • Host forum events to showcase experimental studies conducted in funding organisations
  • Organise ‘Sprint’ events to develop institutional capacity to design and carry out experiments
  • Design experiments ourselves and carry them out across multiple funding organisations, simultaneously where feasible

Within AFIRE, we are currently catalysing, conducting or evaluating experiments on distributed peer review, partial randomisation, AI-assisted reviewer allocation, and desk rejection.

Good funder experiments take rigour and imagination. It’s exciting to be part of a project which is able to draw on both, from across the RoRI network of expertise. Together we’ll be able to support more experiments and better use of experimental evidence to improve research funding.

Tom Stafford, project lead

Project team

(RoRI) – Project lead

Tom Stafford

Senior Research Fellow

Senior Research Fellow, RoRI and Professor, University of Sheffield

Tom Stafford is Professor of Cognitive Science at the University of Sheffield’s School of Psychology, where his work applies insights from neuroscience, experimental design, and data science to the study of learning, decision-making, and bias.

Until August 2026, he is on secondment at RoRI, and from 2026 to 2028, he will hold a Visiting Professorship in the Department of Computer Science and Technology at the University of Cambridge.

As a Senior Fellow at RoRI, Tom leads AFIRE (Accelerator For Innovation & Research Funding Experimentation), a programme dedicated to catalysing high-quality evidence on research processes.

His expertise in metaresearch is further reflected in his role on the PsyArXiv Scientific Advisory Board, where he chairs the Public Engagement Subcommittee.

Previously, Tom served as the University Research Practice Lead for the University of Sheffield (2020–2025), where he represented the institution in the UK Reproducibility Network (UKRN) and chaired the national UKRN Institutional Leads group. An experienced educator, he also designed and led Sheffield’s MSc in Psychological Research Methods with Data Science.

He shares regular updates on his work and writing projects via his Substack, tomstafford.substack.com.

(RoRI, UCL) – Project co-lead

Peter Kolarz

Head of Programmes

Head of Programmes, RoRI 

Pete is part of the core leadership team at RoRI. Current roles include leading the oversight of RoRI’s project portfolio and supporting leadership of the AFIRE and AGORRA projects. Pete is also involved in scoping and designing RoRi’s future portfolio of projects and programmes. 

Formerly a principal consultant at Technopolis, Pete has 12 years’ experience in the field of science and research policy, having managed or directed over 30 projects and worked as a researcher on many more. Highlights from this body of work include a major study on 38 modifications to peer review for grant funding, evaluating UKRI’s research funding response to Covid-19 (process and impact), a peer-learning initiative on partnered research programmes for the GRC, and two major impact assessments for the European Social Survey (ESS). Beyond the above, Pete has worked for clients including the Swiss National Science Foundation, the Research Council of Norway, Wellcome, the Academy of Finland, the Royal Society, the Human Frontier Science Program, and the Austrian Science Fund. 

Pete is a researcher, project manager, communicator and coach, with ample experience of sharing their expertise at several academic and professional conferences and at organisations including the OECD, the GRC, the German Data Forum (RatSWD) and the UK’s House of Lords. They hold a PhD in sociology from the University of Sussex and subsequently published a book on new directions for centre-left politics in the age of globalisation. Pete is a fellow of the Royal Society of Arts, an associate fellow of the Higher Education Academy/ Advance HE and a fluent speaker of English and German. 

Albert Bravo-Biosca

(IGL, Nesta)

Theodore Hodapp

(Gordon and Betty Moore Foundation) – Co-Chair

Stian Westlake

(UKRI) – Co-Chair

(RoRI, UCL)

Josie Coburn

Research Fellow

Research Fellow, RoRI

Josie Coburn is a Research Fellow in Metascience at RoRI at the Department of Science, Technology, Engineering and Public Policy (STEaPP) at University College London (UCL). She is an interdisciplinary social scientist with a portfolio of interests including:

  • Metascience (research on research / science of science), including how research is funded, organised, practiced and evaluated, and research and innovation funding experimentation;
  • The desirability and feasibility of targeting research to address societal needs, how and why research directions evolve over time (e.g. serendipity, failure, and other aspects of research processes and research environments);
  • Opening up decision-making in science, technology and innovation, and the appraisal and evaluation of research and innovation policies and strategies.

Josie works in a range of fields including health and biomedicine, energy and sustainability, and AI and ICTs. She uses a variety of methods including qualitative, scientometric, and hybrid methods. For example, Josie is skilled in Multicriteria Mapping (MCM), a hybrid quantitative-qualitative method for opening up complex decision making by mapping different perspectives on a range of options and taking into account uncertainty.

Before joining RoRI, Josie worked at the Science Policy Research Unit (SPRU) at the University of Sussex as a Research Fellow. She has a PhD in Science and Technology Policy Studies funded by the ERC; MSc’s in Public Policies for Science, Technology and Innovation funded by the ESRC, and in Evolutionary and Adaptive Systems funded by the EPSRC; and a BA in Artificial Intelligence. She also has 4 years’ experience as an Analyst Programmer and Systems Analyst in large and small ICT firms.

Cindy Lopez-Bento

(KU Leuven)

Ines Rombach

(RoRI, University of Sheffield)

(RoRI, University of Sheffield)

Anna Butters

Research Fellow

Research Fellow, RoRI and Research Associate, University of Sheffield 

Anna is a Research Associate at the University of Sheffield. She joined RoRI in 2024 to work with Tom Stafford and Stephen Pinfield on the Distributed Peer Review project.

Her research interests include research improvement, data communication, researcher wellbeing and research evaluation. With a background in psychology, she is interested in applying psychological theory and approaches to understand and improve research practices and systems. 

Anna completed her PhD in Psychology (2025), having previously completed an MA in Social Research (2021) and BSc in Psychology with Foundation Year (2020).

(RoRI, University of Sheffield)

Melanie Benson Marshall

Research Fellow

Research Fellow, RoRI and Research Associate, University of Sheffield

Melanie is a Research Associate in the School of Information, Journalism, and Communication at the University of Sheffield. Her PhD research focused on migration and information behaviour. Her research interests include open science and research cultures, library and information science, and the use of visual and creative methods. She previously worked for a major academic publisher in acquisitions and project management, specialising in engineering and computer science. 

She joined RoRI in 2024 to support our work on Distributed Peer Review with Stephen Pinfield, Tom Stafford, and Anna Butters.

(RoRI, UCL)

James Wilsdon

Executive Director

Executive Director, RoRI and Professor of Research Policy at University College London (UCL)

James is one of the founders of RoRI and has been its director since 2019. He is also Professor of Research Policy at University College London (UCL), based in its Department of Science, Technology, Engineering & Public Policy (STEaPP). Since the late-1990s, as a researcher, writer, adviser and campaigner, James has worked at the heart of science and research policy in the UK and internationally. In addition to academic posts at the universities of Sheffield, Sussex and Lancaster, he has worked in think tanks, NGOs and as director of science policy for the Royal Society, the UK’s national academy of sciences. 

In his work, James has advanced concepts such as upstream public engagementscience diplomacyresponsible metrics and responsible research assessment; and he has co-founded or chaired initiatives such as People & Planet; the Campaign for Social ScienceInternational Network for Government Science Advice (INGSA)UK Forum for Responsible Research Metrics; and the Research on Research Institute (RoRI).

In 2014, James was asked to chair an independent UK government review of research metrics, published as The Metric Tide. He subsequently chaired a European Commission expert group on Next Generation Metrics, and in 2022, revisited these debates with colleagues in Harnessing the Metric Tide.

In 2015, he was elected a Fellow of the UK’s Academy of Social Sciences, and in 2022, of the International Science Council.

(RoRI)

Stephen Curry

Senior Strategic Advisor

Senior Strategic Advisor, RoRI

Stephen has long-standing interests in many different aspects of research, including funding, assessment, equity, culture and publication. He has written extensively on these issues in academic papers and book chapters, in the Guardian, and on his Reciprocal Space blog. Stephen was a founder member of the campaign group Science is Vital in 2010 and served on the board of the Campaign for Science and Engineering from 2012-18. In Autumn 2023 he reached the end of his six-year term as chair of the steering group of the San Francisco Declaration on Research Assessment (DORA), an organization he has helped to position as a singularly influential force for the promotion of research assessment reform worldwide. 

Stephen was a member of the UK government-convened group which in 2015 produced The Metric Tide report on the use of metrics in research assessment and, most recently, was a co-author of the 2022 Harnessing the Metric Tide report, which looked at options for implementing responsible assessment practices within REF 2028. He has served in an advisory or review panel capacity for a range of UK and international organisations including LERUERCJISC,UKRNSNSF and FWF.

Alongside these interests, Stephen has been an academic at Imperial College since 1995, forging a research career as a structural biologist investigating molecular aspects of protein-drug interactions and on the replication of RNA viruses. Latterly, he has served as Imperial’s first Associate Provost for Equality, Diversity and Inclusion (2017-23) and is now a College Consul, in which capacity he remains closely involved in the research and educational mission of the university.

James Phipps

(IGL, Nesta)

Partner organisations

Wellcome

Wellcome

A global charitable foundation supporting science to solve the urgent health issues facing everyone. They work with policy makers, run advocacy campaigns, and form partnerships with other organisations to ensure everyone benefits from advances in health science. 

Luxembourg National Research Fund (FNR)

The Luxembourg National Research Fund (FNR) is the main funder of research activities in Luxembourg. We invest public funds and private donations into research projects in various branches of science and the humanities, with an emphasis on selected core strategic areas. Furthermore, we support and coordinate activities to strengthen the link between science and society and to raise awareness for research. We also advise the Luxembourg government on research policy and strategy.

Innovation Growth Lab (IGL)

The Innovation Growth Lab (IGL) is a global policy lab that drives policy impact through experimentation, evidence and data.

IGL aims to increase the impact of innovation and productivity policies through novel ideas, experimentation, data and evidence.

To do so, we bring together policymakers, researchers, practitioners and funders to address key policy challenges and contribute to more productive, inclusive and sustainable economies.

Interest in experiments is growing, but different research funders are at different stages of readiness as reported by a RoRI-SNSF-EMBO-IGL workshop in December 2021 (internal report ‘RoRI Funder Lab: improving research funding processes through experimentation’).  AFIRE provides a broad platform accessible to funders at all levels of engagement, combined with structured support to work with funders to increase the speed and ambition of their engagement with experiments. Our aims are to enhance:

  • Awareness of experimental possibilities and findings in the funding ecosystem.
  • Motivation to experiment, and to evaluate the effectiveness of interventions
  • Capacity to experiment, encompassing both technical expertise around issues such as data collection and analysis, as well as an understanding of the frontiers of research on research and innovation funding so interventions can be targeted to the most pressing needs.
  • Scale and reach of experiments. The Accelerator will support getting experiments up and running, with greater ambition in running experiments at larger scale, at greater levels of robustness and in ways that support sharing and dissemination of post-experiment evaluation.
  • The culture of research and innovation funding, across the whole sector, so it is more experimental, creative and evidence-informed.

AFIRE Forum

The AFIRE Forum is an arena for peer exchange and learning about experiments at ideation, design stage or in progress. The first event in this series was held in May 2024. Invite only, and conducted under the Chatham House Rule, these informal discussions provide a safe space in which the real work of experimental approaches can be discussed in detail.

AFIRE Sprints

Sprints are on-demand small-group mini-workshop series tackling a particular issue, question, or experimental stage. Sprints involve more intensive and targeted activity with the intention to achieve substantive progress over a short period of time. They bring together a group of partners who are developing experiments within the same topic area or who are at a similar stage in the experimental process. Each sprint consists of a series of sessions facilitated by our team, with participants expected to work independently and make progress on their projects in-between sessions.

The starting point varies for each sprint, and the approach, format and frequency of meetings is adjusted depending on the identified objective, participants’ needs and their availability. Some sprints may bring together multiple partners developing experiments around a similar challenge (e.g., biases) while others may focus instead on helping partners make progress on a particular experimentation stage (e.g., developing a logic model and seeking user feedback).

While the intention is for sprints to bring together multiple partners in order to benefit from peer learning opportunities, there is the option of having sprints that only include one active partner if it is difficult to align timetables and priorities across multiple partners. 

Experiments

Distributed peer review

A major strand of experimental work in AFIRE has been on distributed peer review (DPR). This intervention involved applicants to a funding scheme also acting as reviewers on the same scheme. RoRI researchers have helped conduct experiments on DPR at the Volkswagen Foundation over the past year, with additional experimentation now in progress.

Findings from this work have been widely publicised, and plans are taking shape for RoRI to repeat the experiment at other funding organisations, and to produce guidance and indicator suites for other funders to replicate the experiment by themselves

Article in Nature on Volkswagen Foundation’s Distributed Peer Review experiment, supported by RoRI researchers

Partial randomisation

As a precursor to AFIRE, RoRI’s earlier work on funding process experimentation focused on partial randomisation. This work resulted in The Experimental research Funder’s Handbook, published in 2023. It showcases experiments with partial randomisation conducted at the Volkswagen Foundation, the Swiss National Science Foundation (SNSF), the Austrian Science Fund (FWF), and the Health Research Council of New Zealand.

We continue to monitor and encourage experimentation with partial randomisation and are working on a core outcomes set for such experiments as a resource for funders interested in designing them.

AI-assisted reviewer selection and allocation

We are engaged in several on-going activities around experimentation on AI-assisted reviewer allocation. A number of Sprints have recently begun around this topic, hopefully leading to experimentation in the future. Through the GRAIL project, we have also collected a range of case studies on various funders’ experiments with AI-assisted reviewer allocation.

We also conducted a live-experiment on the abstract review for the recent Metascience2025 conference, comparing a shadow AI-allocation of abstracts to reviewers with reviewers’ self-reported suitability to each abstract following manual allocation.

The case studies and the Metascience2025 experiment are due to be published in early 2026.

Desk Rejection

Desk Rejection is the practice of using internal staff expertise at the funder to identify funding applications which are unlikely to be successful. Adopting desk rejection is an innovation that offers considerable efficiency savings for both funders and reviewers.

So far, the knowledge and expertise of funders’ grant management/administrative staff (and the potential benefits that this expertise may hold) is a critically under-researched topic. This experiment will contribute to filling that gap.

We are setting up a multi-funder shadow experiment, meaning that the funders’ business-as-usual continues, while an alternative process with no real-world consequence (the ‘shadow’ process) is also conducted in parallel, resulting in direct outcomes comparison between the real and shadow processes. The research question we seek to answer is: How accurately does desk rejection correctly categorise rejection via external review?

The AFIRE was launched in 2024 and will run until the end of 2027.


Applicants as reviewers: a guide to distributed peer review

A practical guide for funders exploring or refining Distributed Peer Review, with checklists, setup tips, and lessons from real-world trials


Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations

Paper presenting a mixed-methods evaluation of an experiment with distributed peer review (DPR) carried out by the Volkswagen Foundation


Related publications

Subscribe on Substack

We publish timely essays, reflections and signals from across metascience, research policy and reform. Expect original thinking, reactions to live debates, and practical insights from people studying research.