EFFORT
Estimating Funding Friction: Outcomes, Resource cost and Time
Filter
What are you looking for?
Type
Tags
Sort
Undisciplined
Future models of funding and evaluating transdisciplinary research [completed]
Portfolios
Research funding landscape analysis [completed]
Peer Review
RoRI Atlas of Peer Review [completed]
PATHWAYS
Advancing Research on Research Use to Enhance Positive Impact
Narratives
The uses and evaluation of researchers’ narrative CVs
MetaROR
Translating Publish-Review-Curate outputs into actionable signals for research funders: A MetaROR Study
Matthew
A study of cumulative advantages in funding allocation [completed]
GRAIL
Getting responsible about AI and machine learning in research funding and evaluation [completed]
Funder Data Platform
A Secure Collaboration Hub for Metascience
Criteria
Exploring gender inequality in research funding [completed]
AGORRA
A global observatory of responsible research assessment
Summary
AGORRA (A Global Observatory of Responsible Research Assessment) is a collaboration between research funders, evaluation agencies and meta-researchers which aims to generate comparative data, evidence and analysis to support and accelerate responsible research assessment (RRA). With a specific focus on national-level assessment frameworks, it also aims to inform and accelerate the broader reform and transformation of research assessment systems, supporting and complementing global initiatives like CoARA (The Coalition for Advancing Research Assessment) and DORA (Declaration on Research Assessment). The project consists of three workstreams:
- Collaboratively assessing the possibilities for next-generation research assessment, including the use of novel indicators, dashboard and the feasibility of real-time research assessment to replace burdensome one-off national assessment exercises.
- Populating the Atlas of Assessment, our observatory of national research assessment systems aiming to provide the first global overview of such systems, enabling tracking and comparison of emerging practices, evolutions, convergence, divergence and effectiveness.
- Mapping and defining the indicators of research culture and environment at a global level.
Project team
(RoRI, CWTS) – Project lead
(RoRI, CWTS) – Research Fellow
(RoRI, UCL) – Executive Director, RoRI
(RoRI, UCL) – Research Fellow
(RoRI, formerly NIFU, Norway) – Senior Visiting Fellow
(CONICET, Argentina) – Research Fellow
(RoRI, CWTS) – Senior Research Fellow
(RoRI) – Senior Research Fellow
(RoRI, CWTS) – Co-chair, RoRI
Partner organisations
The project in detail
Strand 1: Next-generation research assessment
Much interest – and in some instances hype – currently surrounds the ‘next generation’ of research metrics. Examples include new approaches for measuring novelty and disruptiveness, open science practices, societal impact, and methods that harness AI and emerging open data sources. In light of such developments, there is a strong case to revisit and update our understanding of metrics use for research assessment.
The fundamental research question underpinning this work is: Can research indicators and dashboards be developed for national assessment systems that harness promising advances in indicator methodologies and data, while also upholding principles of responsible metrics and responsible research assessment more broadly? This includes ensuring the diversity of research cultures is respected, voices of a diverse range of stakeholders are consulted, and unintended consequences are anticipated.
What are the possibilities and limitations for metric use in different kinds of assessment systems, with varying purposes and characteristics? For instance, how do considerations differ between assessments where funding and accountability are key purposes, versus assessments that produce strategic advice?
The Next Generation Research Assessment workstream tackles these timely questions through a mixed-methods, co-produced partnerships between research funders and meta-researchers. Crucially, we are placing our focus on the socio-technical, not just the technical dimensions of new indicators and data sources.
Strand 2: New geographies of research assessment
National systems for research assessment have gained increasing prominence in recent years. These systems are variously used to allocate institutional research funding, to gather intelligence about a country’s research strengths, and to induce behaviour change in the sector. These may pertain to enhancing research productivity or focusing research on particular thematic areas, but also often focus on tenets of RRA as expressed most notably through CoARA (Coalition for Advancing Research Assessment).
Besides a range of different aims, national research assessment systems also use a range of different methods and approaches and focus on different levels of assessment (e.g. institutions, department or individuals). As this diversity of systems increases, so does the need to systematically map them at a global level, to track evolution, convergence and divergence, emerging trends, and assess what works, and in what contexts.
Much of the literature on national research assessment focuses only on a small number of countries, typically from the global north. But to fully understand trends, convergences, divergences, and how different systems and contexts have different needs, a far more comprehensive perspective is required.
The New Geographies of Research Assessment work strand addresses this need. It is a continuation of the AGORRA project, using the Atlas of Assessment – a free, open-access tool that maps research assessment systems worldwide. Launched in May 2025, the Atlas initially included 13 tester countries and continues to expand.
The New Geographies of Research Assessment work strand aims to populate the Atlas as far as possible. The ambition is for the Atlas to become a permanent tool with the fullest possible country coverage and regular updates to all countries included, contributing to a more complete and inclusive picture of research assessment systems.
Strand 3: Indicators of research culture and environment (IRCE)
There is an increasing interest in monitoring and evaluating research cultures and environments as part of national, programme and institutional assessment processes. The RRA agenda aims to expand concepts of quality and excellence with the goal of creating incentives for improving research cultures and environments. This has led to a demand for relevant, trusted and reliable indicators that can provide evidence for monitoring and evaluation systems (the UK being just one recent example).
This strand of AGORRA provides a global assessment of the prospects for research culture indicators. This includes three components:
- Providing a landscape view of qualities for which indicators would be valuable
- A survey of indicators and forms of evidence in use
- A practical review of implementation possibilities including a longitudinal evaluation of candidate indicators.
The workstream is led by the Curtin Open Knowledge Initiative in collaboration with CWTS and the core RORI team.
The workstream has identified the diversity of issues that are tackled globally under the heading of culture and will provide a critical analysis of the qualities of possible indicators. This will include advice on which indicators are appropriate for incorporation into responsible research assessment practices. It will explicitly identify indicators that are inappropriate for application within responsible research assessment systems as well as identifying if and how such indicators might be made fit for purpose over time.
Project timeline
AGORRA runs from early 2023 to mid 2028. There are several core planned milestones and outputs, as well as a range of follow-on options to ensure sustainability and optimal use of results.
Outputs
Building on the influential 2020 report and subsequent action plans, the GRC’s 2025 global survey, developed in collaboration with RoRI, draws insights from 50 funding agencies to map the current landscape of RRA
- The RoRI Atlas of Assessment
- A new typology of national research assessment systems:continuity and change in 13 countries (RoRI Working Paper No.15)
- A previous AGORRA workstream (completed May 2025) mapped the use of process modifications in grant funding via a survey of funders participating in the Global Research Council, in order to understand changing approaches to assessment at the national level: Transforming Assessment: the 2025 Global Research Council survey of funder approaches to responsible research assessment
- Forthcoming (2025): Comparative work on indicators for research culture, which will be an important element of the next UK REF, and are receiving more emphasis in a number of assessment frameworks worldwide.
- AGORRA was the focus of a special session at the STI 2023 conference in Leiden from 27-29 September 2023
Other related outputs from the AGORRA team include:
- The emerging shape of REF 2028 – RoRI open forum, co-hosted with UCL, Foundation for Science and Technology (FST) and Research England, 5 July 2023.
- Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment
- The rise of ‘responsible metrics’ as a professional reform movement: A collective action frames perspective
- REF 2028: A clap for the Frap
- The emerging shape of REF 2028 – RoRI open forum, co-hosted with UCL, Foundation for Science and Technology (FST) and Research England, 5 July 2023.
The changing role of funders in responsible research assessment: progress, obstacles and the way ahead (RoRI Working Paper No.3)
Related initiatives
CoARA (Coalition for Advancing Research Assessment) continues to gather support. The tally of organisational signatories to its underpinning Agreement on Reforming Research Assessment is rising by dozens each week and is now over 800.
In May 2023, DORA (The Declaration on Research Assessment)—on whose foundations CoARA builds—marked its 10th anniversary with a series of workshops around the world. And at a national level, we have seen a sharper focus on these agendas in light of ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand, Sweden and the UK.
A 2023 report on The Future of Research Evaluation by the InterAcademy Partnership, Global Young Academy and International Science Council, provides a guide to the waves of reform visible across assessment systems worldwide.
In China, there has been a renewed drive to break the influence of the “four-onlys” ( sì wéi or 四唯 in Mandarin) — shorthand for a reliance on “only papers, only titles, only education background, and only awards” to determine funding, hiring and promotion of researchers.
This project builds on earlier contributions of RoRI team members to high profile initiatives in this area, such as the Leiden Manifesto, The Metric Tide and Next Generation Metrics. In 2020 RoRI partnered with DORA, UK Research and Innovation (UKRI) and the Global Research Council (GRC) to produce a working paper in support of the GRC Virtual Conference on Responsible Research Assessment
Besides those listed here, AGORRA also intends to support and collaborate with international networks and organisations at the forefront of these debates, including the International Science Council and its Centre for Science Futures, and ENRESSH (European Network for Research Evaluation in the Social Sciences and Humanities), which aims to improve evaluation procedures to take into account the diversity and wealth of SSH research.
