Summary

Peer review performs a vital and expanding role across the research system: in assessing funding proposals; registered reports (and other research designs and protocols); research outputs (including journal articles, proceedings, chapters, books, preprints and datasets); and in contributing to research evaluation exercises, academic appointment and promotion processes. The peer review system is under increasing pressure and there are major challenges around quality, equity and transparency. Analysis of peer review to date has tended to focus on particular domains in isolation (e.g. publishing or grant funding), instead of taking a holistic approach to the question of how peer review might evolve and be improved. The Peer Review project aims to fill this gap by taking a system-wide view of peer review and identifying the key features of peer review across different domains, considering how synergies and efficiencies might be achieved. The project is producing an atlas of peer review and developing various scenarios of what the future of peer review may be, enabling different actors in the system to develop their roles.

In recent years, there has been increasing innovation in peer review, often in response to well-documented problems, such as system overload, bias, and lack of transparency. However, analyses of peer review have tended to focus on particular domains in isolation (e.g. purely on journals or funding proposals), instead of taking a more holistic approach to the question of how peer review might evolve and be improved. Opportunities to innovate and reform peer review and evaluation across different domains may not have been fully explored.

To help fill this gap, we will produce a RoRI Atlas of Peer Review and use this to generate a series of scenarios for the future of peer review. The Atlas will involve a systematic mapping of different domains, actors, objects, criteria, technologies, workflows, functions and economic models. The resulting scenarios will be designed to facilitate discussion between different groups of their possible roles in shaping future trajectories and possibilities for peer review. 

Alongside the Atlas, this project will involve a live experiment in peer review, through the creation and launch of MetaROR (MetaResearch Open Review): a new publication and peer review platform. Developed in close partnership with our friends at the Association for Interdisciplinary Meta-research and Open Science (AIMOS) the MetaROR platform will take an innovative approach to peer review of research outputs in the field of research on research, or meta-research, and will enable the project to test many of the principles and approaches we identify through our broader analysis of the peer review landscape.

Findings from our pilot phase were summarised in this article for the LSE Impact blog on ‘four schools of thought in reforming peer review’. We also published a major report on ‘Scholarly Communications in Times of Crisis’, which explored the responses of the system to the effects of COVID-19.

Project team

Stephen Pinfield, Senior Research Fellow, RoRI and University of Sheffield

Ludo Waltman, Senior Research Fellow, RoRI and CWTS-Leiden

Kathleen Gregory, Research Fellow, RoRI and CWTS-Leiden

Partners and steering group

The Peer Review working group includes the following RoRI partners: 

Australian Research Council 

Canadian Institutes of Health Research (CIHR)

FWF–Austrian Science Fund

The Gordon and Betty More Foundation

UK Research and Innovation (UKRI)

The Social Sciences and Humanities Research Council of Canada (SSHRC)

Volkswagen Foundation

To deepen our engagement with peer review in the scholarly publishing context, we also hope to partner with organisations and initiatives committed to innovation in this arena. Examples include: cOAlition S; Open Research Europe; and ASAPbio, as well as AIMOS and our other partners in the MetaROR platform.