Summary

AGORRA (A Global Observatory of Responsible Research Assessment) is a collaboration between research funders, evaluation agencies and meta-researchers across 14 countries which aims to generate comparative data, evidence and analysis to support and accelerate responsible research assessment (RRA). With a specific focus on national-level assessment frameworks, it also aims to inform and accelerate the broader reform and transformation of research assessment systems, supporting and complementing global initiatives like CoARA (The Coalition for Advancing Research Assessment) and DORA (Declaration on Research Assessment). The project consists of three workstreams:

• Creating and populating an observatory of national research assessment systems, enabling tracking and comparison of emerging practices, evolutions, convergence, divergence and effectiveness;
• Mapping the use of process modifications in grant funding via a survey of funders participating in the Global Research Council, in order to understand changing approaches to assessment at the national level;
• Mapping and defining the concept of research culture and environment at a global level.

---

Project team

• Alex Rushforth, Project lead, CWTS
• Peter Kolarz, Project lead, RoRI
• James Wilsdon, Executive Director, RoRI
• Gunnar Sivertsen, Nordic Institute for Studies of Innovation, Research and Education, Norway
• Cameron Neylon, Researcher, Curtin University
• Claire Fraser, Research England-UKRI
• Amanda Kvarven, Researcher, RoRI/UCL
• Moumita Koley, Researcher, DST, Indian Institute of Science (ISC), India
• Laura Rovelli, Researcher, CONICET, Argentina
• BVE Hyde, Visiting Fellow, RoRI

Partner organisations

• Research England/UKRI
• Australian Research Council 
• ANVUR (Agenzia Nazionale Di Valutazione Del Sistema Universitario E Della Ricerca)
• Evaluation Research Centre, Chinese Academy of Sciences (CAS), Beijing
• Dutch Research Council (NWO)
• The National Research Foundation of South Africa (NRF-SA)
• Canadian Institutes of Health Research (CIHR)
• Social Sciences and Humanities Research Council (SSHRC)
• Swedish Research Council (SRC)
• Research Council of Norway (RCN)
• Volkswagen Foundation

The project in detail

Strand 1: observatory of national research assessment systems

National systems for research assessment have gained increasing prominence in recent years. These systems are variously used to allocate institutional research funding, to gather intelligence about a country’s research strengths, and to induce behaviour change in the sector. These may pertain to enhancing research productivity or focusing research on particular thematic areas, but also often focus on tenets of RRA as expressed most notably through CoARA (Coalition for Advancing Research Assessment).

The AGORRA observatory accomplishes this task. The observatory website is now at prototype stage with information from project partners in 13 countries now being added. Following further testing, the observatory will be made public in early 2025. The ambition is to then populate it with information on as many countries as possible, including countries from the global south, which have been neglected in most existing literature on national research assessment systems.

Strand 2: The GRC RRA Survey

In recent years, there has been a proliferation in modifications to grant funding processes. Increasingly, research funders depart in at least some of their project and fellowship funding from the established ‘standard’ process of submission by 2-3 external academic reviewers followed by a review panel judgement.

Following RoRI’s 2020 survey of Global Research Council participating organisations, this strand of AGORRA is running a new and expanded global survey of research funders. This is creating an up-to-date and comprehensive overview of current practices: what interventions to grant funding processes are being practiced widely already? What is generally perceived to be working well (and what not so well)? Are there emerging convergences/divergences in global practice?

Building on the resulting dataset, the team will produce the first global overview of which grant funding interventions are widely practiced, which ones are not, and what benefits and challenges funders have experienced. This survey also captures for the first time a global picture of AI-use in research funding processes. 

Beyond these descriptive findings, the team will be able to systematically determine ‘what works’ by identifying useful comparators from the survey data and designing quasi-experimental testing of interventions and policies. By making an anonymised dataset available to participants, this work strand will also enable participant organisations to conduct their own analysis.

Descriptive findings from the survey will be presented in a GRC report at the GRC annual summit in May 2025. This will be followed by a phase of more in-depth analysis and a number of subsequent academic publications and policy briefs.

Strand 3: Research culture

There is an increasing interest in monitoring and evaluating research cultures and environments as part of national, programme and institutional assessment processes. The RRA agenda aims to expand concepts of quality and excellence with the goal of creating incentives for improving research culture and environments. This has led to a need for relevant, trusted and reliable indicators that can provide evidence for monitoring and evaluation, exemplified by recent efforts to define ‘People, Culture and Environment’ indicators for the UK’s REF.

This strand of AGORRA will provide a global assessment of the prospects for research culture indicators. This includes three components: providing a landscape view of qualities for which indicators would be valuable, a survey of indicators and forms of evidence in use, and a practical review of implementation possibilities including a longitudinal evaluation of candidate indicators. The workstream will be led by the Curtin Open Knowledge Initiative in collaboration with CWTS and the core RORI team.

In its final stages, the workstream will provide a critical analysis of the qualities of possible indicators, including advice on which ones are appropriate for incorporation into responsible research assessment practices. It will explicitly identify indicators that are inappropriate for application within responsible research assessment systems as well as identifying if and how such indicators might be made fit for purpose over time.

---