Guest Post: Peer review in transition?

This guest post is by members of the Research on Research Institute (RoRI) team: Professor Ludo Waltman (Leiden University), Professor Stephen Pinfield (University of Sheffield), Dr Wolfgang Kaltenbrunner (Leiden University) and Dr Helen Buckley Woods (University of Sheffield)

Several OASPA members and other scholarly publishers are experimenting with new approaches to peer review. The Research on Research Institute (RoRI) is working together with publishers to build an inventory of innovations in peer review. In this post, the RoRI team introduces this project and invites you to contribute.

The functions of peer review in scholarly publishing have always been multiple, including the aim of improving the quality of a colleague’s work, but also that of certifying scientific knowledge and determining eligibility to be published (Csiszar, 2018; Fyfe et al., 2017; Horbach & Halffman, 2018). Spurred on by the overall growth of scientific activity and increasing competition for publishing space since the mid-20th century, questions about efficiency and strain on the invisible economy of peer review labour have become more prominent. In addition, it is ever contentious how well researchers, or any person for that matter, can use their judgement in a consistently fair and unbiased manner.

In recent decades new innovations in peer review have been developed to address issues of bias and inefficiency (Burley, 2017). These innovations are multifarious, but many of them relate to openness of peer review, reviewer incentives, and technological enhancements, such as the use of artificial intelligence (Barogga, 2020).

With such a flurry of diverse and diffusely distributed innovations there is a need to synthesise this work to understand what has been done, the lessons learned and the best way forward. This synthetic work can be seen in numerous literature reviews which draw together the different types of innovation in this area. For example, drawing on the work of Tennant et al. (2017), Barogga (2020) lists 11 types of peer review innovations (such as ‘de-coupled’ and ‘post-publication’) and 9 future models using various web platforms such as ‘third party review’ and ‘publish first, review later’. It can also be seen in new inventories of peer review initiatives, such as the ReimagineReview registry set up by ASAPbio.

In the Research on Research Institute (RoRI), in which science studies researchers collaborate with research funders and scholarly publishers in joint research projects, we aim to build on this work with the creation of an up-to-date inventory of experiments in peer review (and other related forms of quality control) currently being undertaken by publishers and other organisations active in scholarly publishing. We recognise that many important developments in peer review occur as part of ‘learning by doing’ in the day-to-day work of publishing organisations. While some of these initiatives may be more successful than others, there is probably something we can learn from each of them. We plan to distil shared experiences from these initiatives, which will hopefully foster further innovation in peer review.

Tennant (2018) draws out the circular problem of the lack of persuasive evidence for the uptake of newer forms of peer review, while substantial evidence, enough to change minds and behaviour, will only come through a large enough sample of people taking part in these initiatives. This indicates the importance of bringing together all available evidence on the effects of new approaches to peer review.

To support the development of an evidence base and learn more about developments in peer review, we are gathering examples of innovations through a survey. We’d like to invite OASPA members and other publishers to take part. The survey is available here. We would be grateful if you could complete the survey by 9 April. Should you have any further questions regarding the survey or our project, please don’t hesitate to contact us at s.pinfield@sheffield.ac.uk or w.kaltenbrunner@cwts.leidenuniv.nl .

By providing publishers, scholarly societies, journal editors, and the scientific community at large with a current overview of different peer review approaches, we hope to provide fertile ground for further innovations in peer review.

References

Barroga, E. (2020). Innovative strategies for peer review. Journal of Korean Medical Science, 35(20), e138. https://doi.org/10.3346/jkms.2020.35.e138

Csiszar, A. (2018). The Scientific Journal: Authorship and the Politics of Knowledge in the Nineteenth Century. Chicago: Chicago University Press.

Fyfe, A., Coate, K., Curry, S., Lawson, S., Moxham, N., & Røstvik, C. M. (2017). Untangling academic publishing: A history of the relationship between commercial interests, academic prestige and the circulation of research. St Andrews University. https://doi.org/10.5281/zenodo.546100

Horbach, S. P. J. M., & Halffman, W. (2018). The changing forms and expectations of peer review. Research Integrity and Peer Review, 3(8). https://doi.org/10.1186/s41073-018-0051-5

Burley, R. (2017). Peer review in the 21st century: Openness, experimentation, and integrity. Information Services and Use, 37(3), 259-261. https://doi.org/10.3233/ISU-170850

Tennant, J. P. (2018). The state of the art in peer review. FEMS Microbiology Letters, 365(19). https://doi.org/10.1093/femsle/fny204

Tennant, J. P., Dugan, J. M., Graziotin, D., et al. (2017). A multi-disciplinary perspective on emergent and future innovations in peer review [version 3; peer review: 2 approved]. F1000Research, 6(1151) https://doi.org/10.12688/f1000research.12037.3