Summary

Peer review performs a vital and expanding role across the research system: in assessing funding proposals; registered reports (and other research designs and protocols); outputs (including journal articles, proceedings, chapters, books, preprints and datasets); and in contributing to research evaluation exercises, academic appointment and promotion processes.

In recent years, there has been increasing innovation in peer review, often in response to well-documented problems, such as system overload, bias, and lack of transparency. However, analyses of peer review have tended to focus on particular domains in isolation (e.g. purely on journals or funding proposals), instead of taking a more holistic approach to the question of how peer review might evolve and be improved. Opportunities to innovate and reform peer review and evaluation across different domains may not have been fully explored.

To help fill this gap, we will produce a RoRI Atlas of Peer Review and use this to generate a series of scenarios for the future of peer review. The Atlas will involve a systematic mapping of different domains, actors, objects, criteria, technologies, workflows, functions and economic models. The resulting scenarios will be designed to facilitate discussion between different groups of their possible roles in shaping future trajectories and possibilities for peer review. 

Alongside the Atlas, this project will involve a live experiment in peer review, through the creation and launch of MetaROR (MetaResearch Open Review): a new publication and peer review platform. Developed in close partnership with our friends at the Association for Interdisciplinary Meta-research and Open Science (AIMOS) the MetaROR platform will take an innovative approach to peer review of research outputs in the field of research on research, or meta-research, and will enable the project to test many of the principles and approaches we identify through our broader analysis of the peer review landscape.

Findings from our pilot phase were summarised in this article for the LSE Impact blog on ‘four schools of thought in reforming peer review’. We also published a major report on ‘Scholarly Communications in Times of Crisis’, which explored the responses of the system to the effects of COVID-19.

Project team

Stephen Pinfield, Senior Research Fellow, RoRI and University of Sheffield

Ludo Waltman, Senior Research Fellow, RoRI and CWTS-Leiden

Kathleen Gregory, Research Fellow, RoRI and CWTS-Leiden

Partners and steering group

The Peer Review working group includes the following RoRI partners: 

Australian Research Council 

Canadian Institutes of Health Research (CIHR)

FWF–Austrian Science Fund

UK Research and Innovation (UKRI)

Volkswagen Foundation

To deepen our engagement with peer review in the scholarly publishing context, we also hope to partner with organisations and initiatives committed to innovation in this arena. Examples include: cOAlition S; Open Research Europe; and ASAPbio, as well as AIMOS and our other partners in the MetaROR platform.