Summary

Less than a year after it started rolling, the CoARA (Coalition for Advancing Research Assessment) convoy is gathering speed. The tally of organisational signatories to its underpinning Agreement on Reforming Research Assessment is rising by dozens each week. 

In May 2023, DORA (The Declaration on Research Assessment)—on whose foundations CoARA builds—marked its 10th anniversary with a series of workshops around the world. And at a national level, we’ve seen a sharper focus on these agendas in light of ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand, Sweden and the UK.

A recent report on The Future of Research Evaluation by the InterAcademy Partnership, Global Young Academy and International Science Council, provides a guide to the waves of reform visible across assessment systems worldwide.

Even China is getting in on the act, with a renewed drive to break the influence of the “four-onlys” ( sī wéi or 思维 in Mandarin) — shorthand for a reliance on “only papers, only titles, only education background, and only awards” to determine funding, hiring and promotion of researchers.  

So research assessment is changing, but many would argue still not fast enough. RoRI’s AGORRA project aims to generate comparative data, evidence and analysis to support and accelerate this transformation across national assessment systems. Its aims are fourfold:

• To build and connect flexible research capacity that can be directed towards strengthening the evidence base for research assessment reform;
• To monitor ongoing developments in national and international frameworks and policies for research assessment, and support transfer and scaling of good ideas;
• To foster and facilitate a culture of analysis and experimentation among research actors engaged in the design, delivery and evaluation of assessment processes;
• To build evidence, awareness and engagement in the possibilities, pitfalls and choices being made with new methodologies, technologies and indicators

Across research systems, there is increased sophistication in the range of methods being deployed in assessment processes, and the purposes to which such processes are directed. In countries with some form of national performance-based assessment there has typically been a broadening of scope in recent years, from an initial emphasis on productivity, towards a sharper focus on research quality, and then to wider economic and social impacts. 

Further moves are now underway towards the use of assessment as an accelerant of positive change in research culture – sometimes described as a shift from “summative” to “formative” modes of assessment. This is accompanied by renewed efforts to limit the bureaucracy and burden of evaluation processes.

The emergence of CoARA reflects an evolution of these debates towards implementing, sharing and scaling solutions, within a more expansive agenda of responsible research assessment (RRA). This goes beyond debates over measurement to encompass questions about how to create a healthy work culture, how to promote research integrity, how to move from closed to open scholarship, and how to embed principles of equality, diversity and inclusion across the research community

At the same time, there is a need to underpin values-based agendas for reform of research assessment with robust evidence and debate as to what works, and which types of interventions are most effective, or may have unintended consequences. 

The alignment of international capacity for meta-research with priorities and agendas of RRA is still at a relatively early stage. Comparative data on what works is lacking, as are sustained platforms and infrastructures for shared learning. An observatory enables a group of specialist experts or practitioners to monitor a topic, gather data on an ongoing basis, and synthesise evidence for use by others.  

Our vision is for an enhanced and coordinated effort to build connections and learning across national-level approaches to RRA and research assessment. The funder and system-level perspectives of AGORRA will complement other initiatives, such as CoARA and Project TARA, and accelerate the exchange and testing of good ideas from one system to another. 

These agendas require a nuanced approach that combines cutting edge data analytics with social sciences and humanities perspectives to move beyond simple dichotomies of qualitative versus quantitative analysis. To move forward, we need a shared map that combines global data with deep local expertise and knowledge. 

This project builds on earlier contributions of RoRI team members to high profile initiatives in this area, such as the Leiden Manifesto, The Metric Tide and Next Generation Metrics. In 2020 RoRI partnered with DORA, UK Research and Innovation (UKRI) and the Global Research Council (GRC) to produce a working paper in support of the GRC Virtual Conference on Responsible Research Assessment

Project team

James Wilsdon, Director of RoRI and UCL-STEaPP

Alex Rushforth, Senior Research Fellow, RoRI and CWTS-Leiden

Claire Fraser, senior policy adviser, Research England/UKRI

Partners and steering group

The AGORRA project working group will be chaired by Steven Hill from Research England/UKRI. Ailsa McManus from the Australian Research Council will be vice-chair.

Project partners include: 

• Research England/UKRI
• Australian Research Council 
• ANVUR (Agenzia Nazionale Di Valutazione Del Sistema Universitario E Della Ricerca)
• Dutch Research Council (NWO)
• Swiss National Science Foundation (SNSF)
• Canadian Institutes of Health Research (CIHR)
• Social Sciences and Humanities Research Council (SSHRC)
• Swedish Research Council (SRC)
• Research Council of Norway (RCN)
• Volkswagen Foundation

AGORRA will work with and through a network of researchers who will work closely with our funder and evaluation partners:

AGORRA also intends to support and collaborate with international networks and organisations at the forefront of these debates, including:

• DORA – particularly its Project TARA;
• Global Research Council – where RoRI’s 2020 working paper was used as the basis for a major meeting and the formation of an RRA working group;
• CoARA – where we hope to collaborate with one or more of the new CoARA working groups;
• International Science Council and its new Centre for Science Futures;
• ENRESSH (European Network for Research Evaluation in the Social Sciences and Humanities) — which aims to improve evaluation procedures to take into account the diversity and wealth of SSH research