Matthew
Narratives
Grail
Funder Data Platform
AGORRA
Undisciplined
Peer Review
Portfolios
MetaROR
A global observatory of responsible research assessment
Less than a year after it started rolling, the CoARA (Coalition for Advancing Research Assessment) convoy is gathering speed. The tally of organisational signatories to its underpinning Agreement on Reforming Research Assessment is rising by dozens each week.
In May 2023, DORA (The Declaration on Research Assessment)—on whose foundations CoARA builds—marked its 10th anniversary with a series of workshops around the world. And at a national level, we’ve seen a sharper focus on these agendas in light of ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand, Sweden and the UK.
A recent report on The Future of Research Evaluation by the InterAcademy Partnership, Global Young Academy and International Science Council, provides a guide to the waves of reform visible across assessment systems worldwide.
Even China is getting in on the act, with a renewed drive to break the influence of the “four-onlys” ( sī wéi or 思维 in Mandarin) — shorthand for a reliance on “only papers, only titles, only education background, and only awards” to determine funding, hiring and promotion of researchers.
So research assessment is changing, but many would argue still not fast enough. RoRI’s AGORRA project aims to generate comparative data, evidence and analysis to support and accelerate this transformation across national assessment systems. Its aims are fourfold:
Across research systems, there is increased sophistication in the range of methods being deployed in assessment processes, and the purposes to which such processes are directed. In countries with some form of national performance-based assessment there has typically been a broadening of scope in recent years, from an initial emphasis on productivity, towards a sharper focus on research quality, and then to wider economic and social impacts.
Further moves are now underway towards the use of assessment as an accelerant of positive change in research culture – sometimes described as a shift from “summative” to “formative” modes of assessment. This is accompanied by renewed efforts to limit the bureaucracy and burden of evaluation processes.
The emergence of CoARA reflects an evolution of these debates towards implementing, sharing and scaling solutions, within a more expansive agenda of responsible research assessment (RRA). This goes beyond debates over measurement to encompass questions about how to create a healthy work culture, how to promote research integrity, how to move from closed to open scholarship, and how to embed principles of equality, diversity and inclusion across the research community
At the same time, there is a need to underpin values-based agendas for reform of research assessment with robust evidence and debate as to what works, and which types of interventions are most effective, or may have unintended consequences.
The alignment of international capacity for meta-research with priorities and agendas of RRA is still at a relatively early stage. Comparative data on what works is lacking, as are sustained platforms and infrastructures for shared learning. An observatory enables a group of specialist experts or practitioners to monitor a topic, gather data on an ongoing basis, and synthesise evidence for use by others.
Our vision is for an enhanced and coordinated effort to build connections and learning across national-level approaches to RRA and research assessment. The funder and system-level perspectives of AGORRA will complement other initiatives, such as CoARA and Project TARA, and accelerate the exchange and testing of good ideas from one system to another.
These agendas require a nuanced approach that combines cutting edge data analytics with social sciences and humanities perspectives to move beyond simple dichotomies of qualitative versus quantitative analysis. To move forward, we need a shared map that combines global data with deep local expertise and knowledge.
This project builds on earlier contributions of RoRI team members to high profile initiatives in this area, such as the Leiden Manifesto, The Metric Tide and Next Generation Metrics. In 2020 RoRI partnered with DORA, UK Research and Innovation (UKRI) and the Global Research Council (GRC) to produce a working paper in support of the GRC Virtual Conference on Responsible Research Assessment
James Wilsdon, Director of RoRI and UCL-STEaPP
Alex Rushforth, Senior Research Fellow, RoRI and CWTS-Leiden
Claire Fraser, senior policy adviser, Research England/UKRI
The AGORRA project working group will be chaired by Steven Hill from Research England/UKRI. Ailsa McManus from the Australian Research Council will be vice-chair.
Project partners include:
AGORRA will work with and through a network of researchers who will work closely with our funder and evaluation partners:
AGORRA also intends to support and collaborate with international networks and organisations at the forefront of these debates, including:
Research Team
Claire Fraser
Alex Rushforth
James Wilsdon
AGORRA will run for five years to mid-2028.
The project consortium is co-designing a detailed work programme that builds on and connects individual partner and national system efforts to more systematic analysis and experimentation.
Opportunities and outputs identified to date include:
Other related outputs from the AGORRA team include:
The changing role of funders in responsible research assessment: progress, obstacles and the way ahead (RoRI Working Paper No.3)
Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment.
The rise of ‘responsible metrics’ as a professional reform movement: A collective action frames perspective
REF 2028: A clap for the Frap
The emerging shape of REF 2028 – RoRI open forum, co-hosted with UCL, Foundation for Science and Technology (FST) and Research England, 5 July 2023.