Summary

Peer review performs a vital and expanding role across the research system: in assessing funding proposals; registered reports (and other research designs and protocols); research outputs (including journal articles, proceedings, chapters, books, preprints and datasets); and in contributing to research evaluation exercises, academic appointment and promotion processes. The peer review system is under increasing pressure and there are major challenges around quality, equity and transparency. Analysis of peer review to date has tended to focus on particular domains in isolation (e.g. publishing or grant funding), instead of taking a holistic approach to the question of how peer review might evolve and be improved. The Peer Review project aims to fill this gap by taking a system-wide view of peer review and identifying the key features of peer review across different domains, considering how synergies and efficiencies might be achieved. The project is producing an atlas of peer review and developing various scenarios of what the future of peer review may be, enabling different actors in the system to develop their roles.

Project team

• Stephen Pinfield, Project Co-Lead, University of Sheffield
• Ludo Waltman, Project Co-Lead, CWTS
• Kathleen Gregory, Research Fellow, CWTS
• André Brasil, Research Feloow, CWTS

Partner organisations

• Australian Research Council (ARC)
• Canadian Institutes of Health Research (CIHR)
• Austrian Science Fund (FWF)
• The Gordon and Betty More Foundation
• UK Research and Innovation (UKRI)
• Social Sciences and Humanities Research Council of Canada (SSHRC)
• Volkswagen Foundation

The project in detail

The project is examining the different features of peer review across the four main domains in which peer review takes place: funding, scholarly communication, career advancement, evaluation of units. The funding domain is where research grants are assessed and funding allocated. The scholarly communication domain is where research outputs, including journal papers, conference proceedings and books, are reviewed and published. The career advancement domain covers a variety of activities where individuals apply for roles or memberships to advance their careers, including promotion and tenure and memberships of learned societies. The domain encompassing evaluation of units is where national and assessment of other groupings takes place. To these four domains, the research undertaken by the project team has prompted the addition of a fifth: informal peer review. This domain is where a variety of different assessments take place, often relating to the demands of the four other domains, with more or less formality, including feedback to colleagues before a paper is submitted to a journal and internal grant application competitions before selected grants are submitted to funders.

All the domains of peer review are related and all involve researchers in carrying out reviewing activities. However, review processes both within and across domains are often in practice sealed off from each other. Research can be reviewed multiple times, usually organised by different actors for particular purposes without regard to the overall system. Potential synergies between different processes, or portabilities of different review activities, or efficiencies for the system as a whole have rarely been explored.

The Peer Review project is aiming to achieve a more holistic view of peer review across the system, and has two main objectives:

• Objective 1: To develop a systematic understanding of the different characteristics and goals peer review has in different parts of the research system

• Objective 2:  To identify opportunities to adapt or reshape peer review processes to improve the way the different goals are realised, while recognizing the limited peer review capacity in the research system

To achieve these objectives, the project team is carrying interviews with a wide range of stakeholders in the global research community to understand their perspectives on the current and possible future of peer review. The results of these interviews, combined with analysis of the literature and current practice, will enable the production of an atlas of peer review and creation of scenarios of possible peer review futures. Both will help organisations in the research system to assess and, where appropriate, develop their role to improve the research system in future.

The Peer Review project is working in partnership with two other related RoRI initiatives: MetaROR and the AFIRE distributed peer review (DPR) experiment.

MetaROR (MetaResearch Open Review) is a live experiment in peer review: a new publication and peer review platform for meta-research.

Developed in close partnership with the Association for Interdisciplinary Meta-research and Open Science (AIMOS) the MetaROR platform will take an innovative approach to peer review of research outputs in the field of research on research, or meta-research, and will enable the project to test many of the principles and approaches we identify through our broader analysis of the peer review landscape.

The AFIRE DPR experiment is being carried out with the Volkswagen Foundation and involves comparing processes and outcomes of two different peer review processes on grant applications. The DPR process is an innovative approach to peer review involving applicants reviewing each other’s submissions, a radically different approach compared with the conventional process involving a panel of senior researchers making decisions.

Project timeline

The Peer Review project runs from September 2023 to December 2025

Outputs

In September 2024, we published Peer review in funding organizations: An analytical literature review (RoRI Working Paper No.11), to help us develop a systematic understanding of the goals and characteristics of peer review across domains. We also consider possible points for synergies across domains where peer review takes place and pose next steps for future research.

In the final phase of the project, we are producing two headline outputs:

• A working paper detailing the findings from our extensive interview programme
• A report compiling for the first time a comprehensive cross-domain overview of the challenges of the peer review system, alongside an evidence-based framework and practical recommendations to address them (provisional title: ‘The Future of Peer Review’)

Additional information

The current Peer Review project builds on the work of an early RoRI project exploring peer review in journal publishing. That work included a detailed analysis of scholarly communication during the COVID-19 pandemic, carried out with a range of publishers, ‘Scholarly communication in times of crisis’. Work was also carried out identifying innovations in peer review, resulting the influential ‘four schools of peer review’ analysis described in a number of outputs. 

Findings from our pilot phase were summarised in this article for the LSE Impact blog on ‘four schools of thought in reforming peer review’. We also published a major report on ‘Scholarly Communications in Times of Crisis’, which explored the responses of the system to the effects of COVID-19.