The future of peer review is being pulled in multiple directions

Academic research uses a system of peer review for judging whether research is fit to be published. Yet the peer review system is under mounting pressure having not kept up with the pace of change in the ways in which research is done and communicated. As a result, there’s a growing consensus that it’s no longer fit for purpose. 

Innovating peer review is a hot topic with lots of people trying out new ways to tackle  current drawbacks. However, many of these experiments are pulling in different directions. 

As part of our PEER REVIEW project, we’ve shared a preprint analysing the main thrusts and tensions of innovative peer review activities. You can read it here – Innovating peer review, reconfiguring scholarly communication: An analytical overview of ongoing peer review innovation activities.

Exploring innovations in peer review

To find out more about the ways in which people are trying to change peer review, we designed and shared a survey among publishers, academic journal editors and organisations in the scholarly communication ecosystem. We were particularly interested in picking up new ideas and activities that have not yet been properly documented. 

Instead of measuring uptake or effectiveness, our survey asked questions aimed at uncovering the underlying aim, object and actions of each innovation. 

From 54 respondents we created an inventory of 95 innovations which were then ordered using a new taxonomy framework. Building on recent taxonomies by ASAPbio and the International Association of Scientific, Technical and Medical Publishers, our framework allows us to capture and unpack a wider range of innovations than previous studies, giving a rich insight into current thinking in this fast moving area. 

Our taxonomy framework uses five elements. 

You can see the full survey dataset here. 

More than the usual suspects of peer review innovations  

Our survey identified a wide range of innovations, picking up initiatives that some people might not be aware of or that were not included in previous analyses. For example: 

• Peer reviewing datasets, source codes, digital artefacts and research designs 
• Using patient reviewers 
• Using publishing staff to screen for plagiarism, scope and adherence to guidelines
• Providing language support or code review services
• Different ways reviewers are selected including self-selection and using algorithms to automatically match reviewers with relevant manuscripts 
• Rewarding reviewers by making reviewing work visible or directly offering discounts and greater access to journal content
• Collaborative peer review including cross-reviewer commenting, co-reviewing, crowd review and transferable reviews
• Double-blind peer review 

We noticed two areas with a high level of experimentation – preprint reviews and open or transparent peer review – although the individual initiatives within them differed widely.

You can watch the project team share the survey results in our 2021 Metascience session.  

Innovation is pulling peer review in opposing directions 

Friction is to be expected because innovating peer review is a growing field of work with things happening quickly. Yet innovations don’t seem to be moving towards a coherent set of principles.
Tensions are found between:

• Initiatives aiming to make peer review more efficient and less costly while others aim to promote its rigor, which is likely to increase costs
• Innovations based on a singular notion of “good scientific practice” can be at odds with more pluralistic understandings of scientific quality
• The notion of transparency in peer review directly conflicts with the principle that objective reviews should be anonymous

These fault lines show there isn’t an overall agreement or understanding of what needs to happen to improve peer review, suggesting fundamentally different views on the role of peer review and its strengths and weaknesses. 

More coordination – or at least a greater awareness of what is happening elsewhere – is needed to avoid the success of some innovations directly undermining others. RoRI is one of the best places to explore these tensions as we bring together diverse types of organisations and groups, all passionate about peer review, into an independent space for sharing and collaborating. 

What’s next? 

Read our preprint for the full overview of current innovations in peer review and their potential impact on scholarly communication. We’re sharing our findings with those who completed the survey and have published a complementary paper discussing further innovations identified through a literature review.

We’re deciding what we should do next as part of our PEER REVIEW project. Do you have ideas about what questions we should be asking about the future of scholarly publishing? Would you be interested in exploring the tensions between different approaches to peer review further with us? If yes, tell us by getting in touch with project leads Ludo Waltman and Stephen Pinfield.