A global observatory of responsible research assessment


RoRI and DORA’s 2020 working paper for the Global Research Council

Less than a year after it started rolling, the CoARA (Coalition for Advancing Research Assessment) convoy is gathering speed. The tally of organisational signatories to its underpinning Agreement on Reforming Research Assessment is rising by dozens each week. 

In May 2023, DORA (The Declaration on Research Assessment)—on whose foundations CoARA builds—marked its 10th anniversary with a series of workshops around the world. And at a national level, we’ve seen a sharper focus on these agendas in light of ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand, Sweden and the UK.

A recent report on The Future of Research Evaluation by the InterAcademy Partnership, Global Young Academy and International Science Council, provides a guide to the waves of reform visible across assessment systems worldwide.

Even China is getting in on the act, with a renewed drive to break the influence of the “four-onlys” ( sī wéi or in Mandarin) — shorthand for a reliance on “only papers, only titles, only education background, and only awards” to determine funding, hiring and promotion of researchers.  

So research assessment is changing, but many would argue still not fast enough. RoRI’s AGORRA project aims to generate comparative data, evidence and analysis to support and accelerate this transformation across national assessment systems. Its aims are fourfold:

  • To build and connect flexible research capacity that can be directed towards strengthening the evidence base for research assessment reform;
  • To monitor ongoing developments in national and international frameworks and policies for research assessment, and support transfer and scaling of good ideas;
  • To foster and facilitate a culture of analysis and experimentation among research actors engaged in the design, delivery and evaluation of assessment processes;
  • To build evidence, awareness and engagement in the possibilities, pitfalls and choices being made with new methodologies, technologies and indicators

Across research systems, there is increased sophistication in the range of methods being deployed in assessment processes, and the purposes to which such processes are directed. In countries with some form of national performance-based assessment there has typically been a broadening of scope in recent years, from an initial emphasis on productivity, towards a sharper focus on research quality, and then to wider economic and social impacts. 

Further moves are now underway towards the use of assessment as an accelerant of positive change in research culture – sometimes described as a shift from “summative” to “formative” modes of assessment. This is accompanied by renewed efforts to limit the bureaucracy and burden of evaluation processes.

The emergence of CoARA reflects an evolution of these debates towards implementing, sharing and scaling solutions, within a more expansive agenda of responsible research assessment (RRA). This goes beyond debates over measurement to encompass questions about how to create a healthy work culture, how to promote research integrity, how to move from closed to open scholarship, and how to embed principles of equality, diversity and inclusion across the research community

At the same time, there is a need to underpin values-based agendas for reform of research assessment with robust evidence and debate as to what works, and which types of interventions are most effective, or may have unintended consequences. 

The alignment of international capacity for meta-research with priorities and agendas of RRA is still at a relatively early stage. Comparative data on what works is lacking, as are sustained platforms and infrastructures for shared learning. An observatory enables a group of specialist experts or practitioners to monitor a topic, gather data on an ongoing basis, and synthesise evidence for use by others.  

Our vision is for an enhanced and coordinated effort to build connections and learning across national-level approaches to RRA and research assessment. The funder and system-level perspectives of AGORRA will complement other initiatives, such as CoARA and Project TARA, and accelerate the exchange and testing of good ideas from one system to another. 

These agendas require a nuanced approach that combines cutting edge data analytics with social sciences and humanities perspectives to move beyond simple dichotomies of qualitative versus quantitative analysis. To move forward, we need a shared map that combines global data with deep local expertise and knowledge. 

This project builds on earlier contributions of RoRI team members to high profile initiatives in this area, such as the Leiden Manifesto, The Metric Tide and Next Generation Metrics. In 2020 RoRI partnered with DORA, UK Research and Innovation (UKRI) and the Global Research Council (GRC) to produce a working paper in support of the GRC Virtual Conference on Responsible Research Assessment

Project team

James Wilsdon, Director of RoRI and UCL-STEaPP

Alex Rushforth, Senior Research Fellow, RoRI and CWTS-Leiden

Claire Fraser, senior policy adviser, Research England/UKRI

Partners and steering group

The AGORRA project working group will be chaired by Steven Hill from Research England/UKRI. Ailsa McManus from the Australian Research Council will be vice-chair.

Project partners include: 

  • Research England/UKRI
  • Australian Research Council 
  • ANVUR (Agenzia Nazionale Di Valutazione Del Sistema Universitario E Della Ricerca)
  • Dutch Research Council (NWO)
  • Swiss National Science Foundation (SNSF)
  • Canadian Institutes of Health Research (CIHR)
  • Social Sciences and Humanities Research Council (SSHRC)
  • Swedish Research Council (SRC)
  • Research Council of Norway (RCN)
  • Volkswagen Foundation

AGORRA will work with and through a network of researchers who will work closely with our funder and evaluation partners:

Research partnerLocationLead researchersResearch strengths
RoRI core teamUCL
London, UK
James Wilsdon
RoRI secretariat
RRA policies and practices
Responsible metrics
Centre for Science & Technology Studies (CWTS)Leiden, NetherlandsAlex Rushforth
Ludo Waltman
Ismael Rafols
RRA systems, cultures
Responsible metrics
Dutch SEP
Curtin Open Knowledge Initiative (COKI)Perth, AustraliaCameron Neylon
Lucy Montgomery
Large scale data integration Institutional structures and cultures
Open knowledge
Australian ERA / ERA+
Evaluation of OA/mandates 
School of Information Management, Wuhan UniversityWuhan, ChinaLin ZhangQuantitative studies of science;
Research policy;
Bibliometrics: methods;
China’s research assessment system
SWPS UniversityWarsaw, PolandMarta WróblewskaDiscourse analysis Academic communication Scholarly communication Evaluation policy
Latin American Forum on Research Assessment (FOLEC-CLASCO)Argentina, pan-regionLaura RovelliRRA policies and practices in Latin America and Caribbean. 
Peer-review exercises and transformation labs.
Open Science evaluation.
Inclusion, diversity and equity
Nordic Institute for Studies in Innovation (NIFU)Oslo, NorwayGunnar SivertsenResearch policy, national assessment frameworks, scientometrics, open science evaluation

AGORRA also intends to support and collaborate with international networks and organisations at the forefront of these debates, including:

Credit: Global Research Council’s RRA Working Group

Research Team

Timeline and outputs

AGORRA will run for five years to mid-2028.

The project consortium is co-designing a detailed work programme that builds on and connects individual partner and national system efforts to more systematic analysis and experimentation. 

Opportunities and outputs identified to date include:

  • Building on review processes in countries like Australia, Italy, Sweden and the UK, where there are ongoing debates about the design and reform of national assessment frameworks, influenced by international moves towards RRA reform.
  • Comparative work on indicators for research culture, which will be an important element of the next UK REF, and are receiving more emphasis in a number of assessment frameworks worldwide.
  • Building on the COKI Research Impact Evaluation System, which can be used to model and predict how different design choices for quantitative assessment will play out in future national assessments. This system offers an opportunity to investigate the sensitivity and robustness of quantitative exercises by modelling the entire system.
  • Producing an overview of the recent history of performance based funding systems – from 2010 to the present – and charting the emergence of more formative models.

Other related outputs from the AGORRA team include:

The changing role of funders in responsible research assessment: progress, obstacles and the way ahead (RoRI Working Paper No.3)

Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment.

The rise of ‘responsible metrics’ as a professional reform movement: A collective action frames perspective

REF 2028: A clap for the Frap

The emerging shape of REF 2028 – RoRI open forum, co-hosted with UCL, Foundation for Science and Technology (FST) and Research England, 5 July 2023.