Desk rejection shadow experiment

Summary

The desk rejection experiment is a multi-funder shadow experiment to test the feasibility and effectiveness of using desk rejection in funding application assessment processes 

Desk rejection is the practice of using funder staff expertise and experience to identify and reject applications which are unlikely to be successful 

Rationale: Against the backdrop of growing applications numbers, desk rejection would allow funders to reduce the volume of applications sent out for peer review 

Anticipated results: Co-produced in partnership with RoRI and a global coalition of funders, the results could inform changes to funding delivery practice at a global scale. Results will be reported in aggregate form and individuals participating in the experiment will not be identified 

Project team 

Research Fellow, RoRI

Josie Coburn

Research Fellow

j.coburn@researchonresearch.org

Research Fellow, RoRI

Josie Coburn is a Research Fellow in Metascience at RoRI at the Department of Science, Technology, Engineering and Public Policy (STEaPP) at University College London (UCL). She is an interdisciplinary social scientist with a portfolio of interests including:

  • Metascience (research on research / science of science), including how research is funded, organised, practiced and evaluated, and research and innovation funding experimentation;
  • The desirability and feasibility of targeting research to address societal needs, how and why research directions evolve over time (e.g. serendipity, failure, and other aspects of research processes and research environments);
  • Opening up decision-making in science, technology and innovation, and the appraisal and evaluation of research and innovation policies and strategies.

Josie works in a range of fields including health and biomedicine, energy and sustainability, and AI and ICTs. She uses a variety of methods including qualitative, scientometric, and hybrid methods. For example, Josie is skilled in Multicriteria Mapping (MCM), a hybrid quantitative-qualitative method for opening up complex decision making by mapping different perspectives on a range of options and taking into account uncertainty.

Before joining RoRI, Josie worked at the Science Policy Research Unit (SPRU) at the University of Sussex as a Research Fellow. She has a PhD in Science and Technology Policy Studies funded by the ERC; MSc’s in Public Policies for Science, Technology and Innovation funded by the ESRC, and in Evolutionary and Adaptive Systems funded by the EPSRC; and a BA in Artificial Intelligence. She also has 4 years’ experience as an Analyst Programmer and Systems Analyst in large and small ICT firms.

Projects

Senior Research Fellow, RoRI

Tom Stafford

Senior Research Fellow

t.stafford@sheffield.ac.uk

Senior Research Fellow, RoRI and Professor, University of Sheffield

Tom Stafford is Professor of Cognitive Science at the University of Sheffield’s School of Psychology, where his work applies insights from neuroscience, experimental design, and data science to the study of learning, decision-making, and bias.

Until August 2026, he is on secondment at RoRI, and from 2026 to 2028, he will hold a Visiting Professorship in the Department of Computer Science and Technology at the University of Cambridge.

As a Senior Fellow at RoRI, Tom leads AFIRE (Accelerator For Innovation & Research Funding Experimentation), a programme dedicated to catalysing high-quality evidence on research processes.

His expertise in metaresearch is further reflected in his role on the PsyArXiv Scientific Advisory Board, where he chairs the Public Engagement Subcommittee.

Previously, Tom served as the University Research Practice Lead for the University of Sheffield (2020–2025), where he represented the institution in the UK Reproducibility Network (UKRN) and chaired the national UKRN Institutional Leads group. An experienced educator, he also designed and led Sheffield’s MSc in Psychological Research Methods with Data Science.

He shares regular updates on his work and writing projects via his Substack, tomstafford.substack.com.

Projects
RoRi publications
Applicants as Reviewers: Evaluating the Risks, Benefits, and Potential of Distributed Peer Review for Grant Funding Allocations (RoRI Working Paper No. 17) Applicants as reviewers – a Guide to Distributed Peer Review Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives (RoRI Working Paper No.9) The experimental research funder’s handbook Demand management: findings from a multi-funder workshop Applicants as reviewers: a mixed methods evaluation of the risks, benefits, and potential of distributed peer review for grant funding allocations

Partner organisations 

The project in detail 

Desk Rejection is the practice of using internal staff expertise at the funder to identify funding applications which are unlikely to be successful. Adopting desk rejection is an innovation that offers considerable efficiency savings for both funders and reviewers. Deciding on the feasibility of desk rejection requires insight into its accuracy and error rate. This can be analysed by tracking assessments about application outcomes by internal staff and comparing these assessments to the outcomes under full review. 

This is a shadow experiment, meaning that the funders’ business-as-usual continues, while an alternative process with no real-world consequence (the ‘shadow’ process) is also conducted in parallel, resulting in direct outcomes comparison between the real and shadow processes. The research question we seek to answer is: How accurately does desk rejection correctly categorise rejection via external review? 

In this case, the experiment will involve a small number of call/programme managers providing their assessment of whether each proposal submitted to a funding call will ultimately be successful. These assessments will then be compared with actual application review process outcomes. 

To produce meaningful evidence beyond just a single funder’s case, this experiment will take place across several funding organisations. 

Depending on the outcomes of the analysis, this experiment will either affirm the value and importance of peer review (in the case that internal review cannot correctly categorise actual outcomes), or demonstrate an area of potential efficiencies (in the case that internal review can correctly categorise at least a subset of outcomes). 

More broadly, it will provide insight into the level of experience and expertise required to undertake desk rejection (as these levels are expected to vary among internal staff and with respect to different calls). 

So far, the knowledge and expertise of funders’ grant management/administrative staff (and the potential benefits that this expertise may hold) is a critically under-researched topic. This experiment will contribute to filling that gap. 

This shadow experiment is intended for funding calls that don’t currently use desk rejection. 

Project timeline 

The Desk Rejection Shadow Experiment will run from mid-2025 to mid-2027. 

Outputs 

Anticipated outputs from the desk rejection shadow experiment are: 

Additional information 

The following materials provide more details about the desk rejection shadow experiment: 

Subscribe on Substack

We publish timely essays, reflections and signals from across metascience, research policy and reform. Expect original thinking, reactions to live debates, and practical insights from people studying research.