Summary

Research on research, or metaresearch, is changing fast. In the past five years, researchers from a diverse range of disciplines have contributed to such efforts, and new communities of scholarship and practice have emerged. It is exciting to see fresh energy and methodological innovation being directed to these agendas. But this can also bring challenges, particularly in how research outcomes are communicated and evaluated.

• Siloed organisation. There is limited interaction between different research on research communities (e.g., history of science, innovation studies, philosophy of science, science and technology studies, science of science, scientometrics).
• Limited openness in research practices. Much research in our field is shared only in its final state, typically as an article published in a journal. Moreover, the overall adoption of preprinting, open peer review, data sharing, preregistration, and other open science practices is uneven across meta-research communities.
• Limited accessibility of our literature. In many cases, readers need to pay to read the literature or authors need to pay to contribute to the literature. Those who are unable to pay do not have full access to the literature.
• Pressure on peer review. In this as in every field, peer review is under pressure, delaying the communication of research outcomes and making it difficult to organise reliable processes for evaluating research results.
• Lack of community ownership. Many of the most important journals in our field are (co-)owned by commercial publishers, constraining the freedom that these communities have to decide for themselves how to organise review and publication practices.

These problems are far from unique to our fields. But metaresearch has a particular responsibility to develop and test innovative solutions. To this end, RoRI and the Association for Interdisciplinary Meta-Research and Open Science (AIMOS) are launching a new platform for review and scholarly communication called MetaROR (MetaResearch Open Review). 

Within RoRI’s portfolio, MetaRoR sits within our broader strand of work on Peer Review, as a practical experiment and testbed for some of the ideas being explored in that project. While our plans for MetaROR are still being fine-tuned, the platform will work like this:

• MetaROR will not be a traditional scholarly journal, but a platform that operates according to a publish-review-curate model. This model is getting increasingly popular, especially in the life sciences, where it is used by journals such as eLife and F1000 Research.

• Under MetaROR’s publish-review-curate model, researchers will first publish their work on a preprint server such as MetaArXiv, SocArXiv or OSF Preprints and then submit it to MetaROR. Submissions will be handled by MetaROR editors, who will first perform a basic screening and then assign reviewers on the basis of their fit with a submission. 

• The role of a MetaROR editor is a form of voluntary communal service and will be advertised on a rolling basis on the platform website. Review reports and (optionally) reviewer identities will be published on the MetaROR platform and linked to the article in preprint form. 

• Based on the peer review outcomes, MetaROR will publish an editorial assessment consisting, for instance, of a short summary, contextualization, and brief discussion of review reports. This is the “curate” aspect of publish-review-curate models.

• Research reviewed and curated by MetaROR can still be published in traditional scholarly journals. To streamline this process, MetaROR aims to develop partnerships with journals in adjacent fields of metaresearch. This is similar to the way in which platforms such as Review Commons and Peer Community In are partnering with academic journals.

• MetaROR’s publish-review-curate model will accelerate the communication of scholarly work, since peer review will take place after publication rather than before. The model is also expected to reduce pressures on peer review, since reviews will be used more efficiently. 

• Should authors of an accepted paper choose to submit their work to a journal later on, they can include the reviews that have already been performed for MetaROR. And since reviews will be openly available, there will be more recognition for the efforts of reviewers than is usually the case (making it more attractive for researchers to perform peer review).

• MetaROR aims to stimulate interaction between different research communities, while recognising the value of community-specific norms and research practices. In addition, MetaROR aims to promote the translation of research outcomes to insights that are of direct practical use, for instance those involved in community building or funding of transdisciplinary research.

MetaROR will be owned by the academic community of researchers doing research on science, technology and knowledge making. We will work together with one or more technology providers, but they will not own the platform. Instead, we envision MetaROR to be community-owned.

Project team

Stephen Pinfield, Senior Research Fellow, RoRI and University of Sheffield

Ludo Waltman, Senior Research Fellow, RoRI and CWTS-Leiden  

André Brasil, Research Fellow, RoRI and CWTS-Leiden

Plus colleagues from the Association for Interdisciplinary Meta-Research and Open Science (AIMOS)

Partners and steering group

MetaROR is a joint initiative of AIMOS and RoRI. To build MetaROR into a community-driven collaboration that reflects the rich and growing diversity of metaresearch, we hope to further expand its project and editorial team. We invite anyone interested in contributing to MetaROR’s development and implementation to reach out to us.

Within RoRI, MetaRoR is an applied experiment within our wider project on Peer Review. The working group for this includes the following RoRI partners: 

Australian Research Council 

Canadian Institutes of Health Research (CIHR)

UK Research and Innovation (UKRI)

Volkswagen Foundation

To deepen our engagement with peer review in the scholarly publishing context, we also hope to partner with organisations and initiatives committed to innovation in this arena. Examples include: cOAlition S; Open Research Europe; and ASAPbio, as well as AIMOS and our other partners in the MetaROR platform.