Matthew
Narratives
GRAIL
Funder Data Platform
AGORRA
Undisciplined
Peer Review
Portfolios
MetaROR
AFIRE
Academic research uses a system of peer review for judging whether research is fit to be published. Yet the peer review system is under mounting pressure having not kept up with the pace of change in the ways in which research is done and communicated. As a result, there’s a growing consensus that it’s no longer fit for purpose.
Innovating peer review is a hot topic with lots of people trying out new ways to tackle current drawbacks. However, many of these experiments are pulling in different directions.
As part of our PEER REVIEW project, we’ve shared a preprint analysing the main thrusts and tensions of innovative peer review activities. You can read it here – Innovating peer review, reconfiguring scholarly communication: An analytical overview of ongoing peer review innovation activities.
To find out more about the ways in which people are trying to change peer review, we designed and shared a survey among publishers, academic journal editors and organisations in the scholarly communication ecosystem. We were particularly interested in picking up new ideas and activities that have not yet been properly documented.
Instead of measuring uptake or effectiveness, our survey asked questions aimed at uncovering the underlying aim, object and actions of each innovation.
From 54 respondents we created an inventory of 95 innovations which were then ordered using a new taxonomy framework. Building on recent taxonomies by ASAPbio and the International Association of Scientific, Technical and Medical Publishers, our framework allows us to capture and unpack a wider range of innovations than previous studies, giving a rich insight into current thinking in this fast moving area.
Our taxonomy framework uses five elements.
You can see the full survey dataset here.
Our survey identified a wide range of innovations, picking up initiatives that some people might not be aware of or that were not included in previous analyses. For example:
We noticed two areas with a high level of experimentation – preprint reviews and open or transparent peer review – although the individual initiatives within them differed widely.
You can watch the project team share the survey results in our 2021 Metascience session.
Friction is to be expected because innovating peer review is a growing field of work with things happening quickly. Yet innovations don’t seem to be moving towards a coherent set of principles.Tensions are found between:
These fault lines show there isn’t an overall agreement or understanding of what needs to happen to improve peer review, suggesting fundamentally different views on the role of peer review and its strengths and weaknesses.
More coordination – or at least a greater awareness of what is happening elsewhere – is needed to avoid the success of some innovations directly undermining others. RoRI is one of the best places to explore these tensions as we bring together diverse types of organisations and groups, all passionate about peer review, into an independent space for sharing and collaborating.
Read our preprint for the full overview of current innovations in peer review and their potential impact on scholarly communication. We’re sharing our findings with those who completed the survey and have published a complementary paper discussing further innovations identified through a literature review.
We’re deciding what we should do next as part of our PEER REVIEW project. Do you have ideas about what questions we should be asking about the future of scholarly publishing? Would you be interested in exploring the tensions between different approaches to peer review further with us? If yes, tell us by getting in touch with project leads Ludo Waltman and Stephen Pinfield.