Next steps for research on research: RoRI phase two is underway

On 20th June 2022, we welcomed around 170 participants to the launch of RoRI’s second phase. The event was co-hosted by the Swiss National Science Foundation (SNSF), who welcomed in-person participants in Bern and live streamed a packed afternoon of talks.  

Together, we debated priorities for research on research, shared lessons and highlights from our pilot phase, and looked ahead to RoRI’s next five years. 
You can watch the event here and below we share some of the highlights of the day.

A call to arms: get experimental with research funding
Matthias Egger, SNSF President, opened the event by explaining why so many partners have joined RoRI since its launch in 2019. Partly for self-interested reasons: being part of RoRI can help funders to better align their positions on strategically important topics. Also to strengthen the wider research enterprise: through RoRI, organisations can contribute to the evidence base for national and international research policy.  

“Think of the vast amount of money that goes into research, yet how little goes into ensuring research is done in a good way. RoRI helps us to redress that balance”, said Matthias.
Next we heard from Michael Nielsen, a pioneer of quantum computing and open science, who presented a provocative and engaging manifesto for new models of funding research and the role of meta-science in helping to determine and evaluate interventions.

“Funding shouldn’t be based on what sounds good to politicians, funders or individual scientists but on what will build a better understanding of how the world works”, said Michael. “In order to do that, we need to reliably know what works. I would love to see a scale up of radical funding experiments.” 

RoRI showcase: projects, tools and outputs 
We then heard highlights of what’s been achieved during RoRI’s pilot phase. 

• Funder Data Platform 

A colossal team effort to share data across multiple funders has resulted in the creation of the Funder Data Platform (FDP)—a secure, closed environment for research funders to upload, share and collaboratively analyse funding data.

The platform can securely hold and link a vast amount of data, and this has been used in our CRITERIA project, with early findings around gender differences in funding rates now available, and a full report to follow soon.

Putting the necessary legal and project agreements in place for the FDP was no small feat but the effort has been worth it. By sharing data across multiple funders, we can now see much further than any individual funder alone. In phase two, thanks to a generous 5-year grant from the Novo Nordisk Foundation, the FDP will be developed further in partnership with the Danish Centre for Studies in Research and Research Policy at Aarhus University . 

• Turning lessons learned into practical guidance for experimenting with research funding

Our RANDOMISATION project picked up speed throughout the pilot. We moved beyond simply talking about the need for new ways of allocating research funding to developing practical tips and guidance for how funders can do it themselves, which are presented in our Experimental Research Funder’s Handbook. 

While we took an initial focus on partial randomisation, we’ve also helped funders incorporate an experimental way of thinking in other internal processes, with more organisations looking to build on this progress during phase two.

• New understanding of how peer review responds in times of crisis

Through collaboration with several publishers and infrastructure providers, our PEER REVIEW project provided a data-led insight into the extent to which the scholarly communications community delivered on commitments to open science during the pandemic. Our work also encouraged publishers to take steps forward in their approaches to open data sharing.

• Moving towards FAIR metadata practices

Uptake of the FAIR principles is now becoming more widespread across research. Our FAIRware project is developing the FAIR Workbench, a piece of open-source software that researchers can use to assess their own metadata to ensure it adheres to FAIR standards. With the help of a team from the Stanford Center for Biomedical Informatics Research, we’re on the way to turning existing reporting guides into machine-processable forms. The team are now seeking scientists who can help test the prototype. 

• Transforming ‘excellence’ 

It’s now widely understood that using ‘excellence’ as a criterion in assessing research can be problematic. Our EXCELLENCE project has helped funders to challenge and develop their uses of ‘excellence’.

We found that ‘excellence’ has moved from a matter of fact (“I know it when I see it”) to a matter of concern, which should be debated. By unpacking ways in which organisations now use ‘excellence’, we’ve uncovered three common strategies: patching, pluralising, and transforming. 

What to expect in the next five years
James Wilsdon, RoRI’s director, gave a flavour of what we can expect as phase two gets underway. As well as incorporating RoRI as a nonprofit social enterprise, our next wave of projects will span four themes: 

• experiments with evaluation;
• infrastructures and data sharing;
• research priorities and portfolios; and
• impacts, indicators and culture change.  

“Everyone involved in RoRI has come together because we believe that more robust evidence of how to improve research systems and cultures is needed. And that a collaborative, cross-country model is the best way to test new interventions”, says James. “RoRI is far more than just its outputs. The consortium itself, and how we now work together, is our greatest achievement.” 

To close the afternoon, we heard from a panel of RoRI partners chaired by Katrin Milzow, Head of Strategy at SNSF and RoRI Co-Chair. This explored:  

• How the research on research community is making a difference: for example, providing the insight needed to translate solutions to the reproducibility crisis. An increasing number of organisations are now applying metascience principles to change how they work internally.

• Knowledge gaps and what RoRI could look at next: in particular, involving social sciences and the humanities to help us understand the complex web of political and social ties and influences that affect how research is run. 

Opportunities to get involved
All the speakers agreed that there’s plenty for RoRI to get stuck into over the next five years. No single organisation can change the culture of science, which is why RoRI’s role in convening different stakeholders is important.   

We invite you to join us in shaping RoRI’s agenda for the next five years. To explore partnership and project opportunities, please contact James Wilsdon at hello@researchonresearch.org.

Photo credit: Adrian Moser Photography.