What are you looking for?

AGORRA

A global observatory of responsible research assessment

Summary

AGORRA (A Global Observatory of Responsible Research Assessment) is a collaboration between research funders, evaluation agencies and meta-researchers which aims to generate comparative data, evidence and analysis to support and accelerate responsible research assessment (RRA). With a specific focus on national-level assessment frameworks, it also aims to inform and accelerate the broader reform and transformation of research assessment systems, supporting and complementing global initiatives like CoARA (The Coalition for Advancing Research Assessment) and DORA (Declaration on Research Assessment). The project consists of three workstreams:

  • Collaboratively assessing the possibilities for next-generation research assessment, including the use of novel indicators, dashboard and the feasibility of real-time research assessment to replace burdensome one-off national assessment exercises.
  • Populating the Atlas of Assessment, our observatory of national research assessment systems aiming to provide the first global overview of such systems, enabling tracking and comparison of emerging practices, evolutions, convergence, divergence and effectiveness.
  • Mapping and defining the indicators of research culture and environment at a global level.

Project team

(RoRI, CWTS) – Project lead

Alex Rushforth

Senior Research Fellow

Senior Research Fellow, RoRI and Assistant Professor at CWTS, Leiden University

Alex Rushforth is Assistant Professor at CWTS, Leiden University, the Netherlands. He works in the social studies of science, with particular focus on research evaluation. Alex has longstanding interests in the uses of indicators in research and evaluation settings and his current research interests focus on the emergence, implementation and (un)intended effects of reform policies for research assessment.

Alex is currently coordinator of RoRi’s AgoRRA Project, with James Wilsdon and Claire Fraser.

(RoRI, CWTS) – Research Fellow

Nina Gogadze

Research Fellow

Research Fellow, RoRI and PhD Candidate at CWTS, Leiden University

Nina Gogadze is a PhD candidate at the Centre for Science and Technology Studies (CWTS) at Leiden University. 

Nina is an alumna of the Erasmus Mundus Joint Master’s Degree Programme in Research and Innovation in Higher Education (MARIHE) and holds a master’s degree from the University for Continuing Education Krems and Tampere University.

Her research interests include research assessment reform movements, national research evaluation systems, and the Europeanisation of higher education and science policies in post-Soviet countries.

(RoRI, UCL) – Research Fellow

Seunghyun Lee

Research Fellow

Research Fellow, RoRI

Seunghyun is a Research Fellow in Metascience who joined RoRI in 2025. Her work with RoRI focuses on the AGORRA (A Global Observatory of Responsible Research Assessment) project, contributing to efforts to reform research assessment and strengthen the role of research in society.

Seunghyun is a policy researcher and analyst with expertise in science, technology, and innovation (STI) policy, research assessment, and institutional strategy. Her doctoral research at the University of Manchester explored the positioning of public research institutes within national innovation systems, suggesting a new framework for evaluating research actors through bibliometric and network analysis. This work contributes to emerging debates in metascience around interdisciplinarity, collaboration, and innovation.

Before joining RoRI, Seunghyun worked with leading national and international organisations, including the OECD and Korea Institute of Science and Technology Evaluation and Planning (KISTEP), on STI strategy, capability indicators, and evaluation of public research institutes.

(RoRI, CWTS) – Research Fellow

Judit Varga

Research Fellow

Research Fellow, RoRI and Researcher, Centre for Science and Technology Studies (CWTS), Leiden University

Judit Varga is a Researcher at the Centre for Science and Technology Studies (CWTS) at Leiden University in the Netherlands. She is a social scientist with a doctorate in science and technology studies (STS) (University of Nottingham, 2021) and a background in psychology. Her work focuses on exploring the interactions between quantitative and interpretative methods in social science; the politics of quantification; values in science (evaluation); interdisciplinarity; and science in society.

Her doctoral research provided new methodological insights about combining STS and scientometrics, and explored interdisciplinary interactions in a subfield of computational social science.

As part of RoRI, Judit studies how narrative CVs shape science evaluation with Helen Buckley Woods and Wolfgang Kaltenbrunner.

(RoRI, UCL) – Head of Programmes, RoRI

Peter Kolarz

Head of Programmes

Head of Programmes, RoRI 

Pete is part of the core leadership team at RoRI. Current roles include leading the oversight of RoRI’s project portfolio and supporting leadership of the AFIRE and AGORRA projects. Pete is also involved in scoping and designing RoRi’s future portfolio of projects and programmes. 

Formerly a principal consultant at Technopolis, Pete has 12 years’ experience in the field of science and research policy, having managed or directed over 30 projects and worked as a researcher on many more. Highlights from this body of work include a major study on 38 modifications to peer review for grant funding, evaluating UKRI’s research funding response to Covid-19 (process and impact), a peer-learning initiative on partnered research programmes for the GRC, and two major impact assessments for the European Social Survey (ESS). Beyond the above, Pete has worked for clients including the Swiss National Science Foundation, the Research Council of Norway, Wellcome, the Academy of Finland, the Royal Society, the Human Frontier Science Program, and the Austrian Science Fund. 

Pete is a researcher, project manager, communicator and coach, with ample experience of sharing their expertise at several academic and professional conferences and at organisations including the OECD, the GRC, the German Data Forum (RatSWD) and the UK’s House of Lords. They hold a PhD in sociology from the University of Sussex and subsequently published a book on new directions for centre-left politics in the age of globalisation. Pete is a fellow of the Royal Society of Arts, an associate fellow of the Higher Education Academy/ Advance HE and a fluent speaker of English and German. 

(RoRI, UCL) – Executive Director, RoRI

James Wilsdon

Executive Director

Executive Director, RoRI and Professor of Research Policy at University College London (UCL)

James is one of the founders of RoRI and has been its director since 2019. He is also Professor of Research Policy at University College London (UCL), based in its Department of Science, Technology, Engineering & Public Policy (STEaPP). Since the late-1990s, as a researcher, writer, adviser and campaigner, James has worked at the heart of science and research policy in the UK and internationally. In addition to academic posts at the universities of Sheffield, Sussex and Lancaster, he has worked in think tanks, NGOs and as director of science policy for the Royal Society, the UK’s national academy of sciences. 

In his work, James has advanced concepts such as upstream public engagementscience diplomacyresponsible metrics and responsible research assessment; and he has co-founded or chaired initiatives such as People & Planet; the Campaign for Social ScienceInternational Network for Government Science Advice (INGSA)UK Forum for Responsible Research Metrics; and the Research on Research Institute (RoRI).

In 2014, James was asked to chair an independent UK government review of research metrics, published as The Metric Tide. He subsequently chaired a European Commission expert group on Next Generation Metrics, and in 2022, revisited these debates with colleagues in Harnessing the Metric Tide.

In 2015, he was elected a Fellow of the UK’s Academy of Social Sciences, and in 2022, of the International Science Council.

(RoRI, formerly NIFU, Norway) – Senior Visiting Fellow

Gunnar Sivertsen

Visiting Senior Fellow

Visiting Senior Fellow, RoRI and Research Professor Emeritus, the Nordic Institute for Studies in Innovation, Research and Education (NIFU)

Gunnar Sivertsen is Research Professor Emeritus at the Nordic Institute for Studies in Innovation, Research and Education (NIFU) in Oslo, Norway. Sivertsen’s research contributes to science-based innovation in the development of research policy, evaluation, and funding, and in the use of aggregate indicators for the same purposes.  

(RoRI) – Senior Research Fellow

Cameron Neylon

Senior Research Fellow

Senior Research Fellow, RoRI and Professor of Research Communication at the Centre for Culture and Technology at Curtin University 

Cameron Neylon is Professor of Research Communication at the Centre for Culture and Technology at Curtin University where he is co-lead on the Curtin Open Knowledge Initiative. He is interested in how to make the internet more effective as a tool for scholarship. He writes and speaks regularly on scholarly communication, the design of web based tools for research, and the need for policy and cultural change within and around the research community.

Cameron Neylon is a one-time biomedical scientist who has moved into the humanities via Open Access and Open Data advocacy. His research and broader work focusses on how we can make the institutions that support research sustainable and fit for purpose for the 21st century and how the advent of new communications technology is a help (and in some cases a hindrance) for this.

(DST, Indian Institute of Science (ISC), India)

Moumita Koley

Research Fellow

Research Fellow, RoRI and Senior Research Analyst at DST-Centre for Policy Research, Indian Institute of Science

Moumita Koley is a senior research analyst at the DST-Centre for Policy Research, IISc, Bangalore, India. She also worked as a consultant for the International Science Council’s Future of Scientific Publishing project between March 2023 and June 2024.

Her research interest is in enhancing the accessibility of scientific output, making the research ecosystem more efficient and responsible, and ensuring that research is responsive to local problems.

Moumita holds a Ph.D. in organic chemistry from the Vienna University of Technology in Austria, where she focused on synthesising small molecules that can direct the differentiation of pluripotent stem cells towards heart muscle cells. Later, she became interested in exploring the structure and evolution of science, including various dimensions of open science, such as open access, open research data, and research evaluation.

(CONICET, Argentina)

Laura Rovelli

Research Fellow

Research Fellow, RoRI and Independent Researcher, the National Scientific and Technical Research Council (CONICET) in Argentina

Laura Rovelli is an independent researcher at the National Scientific and Technical Research Council (CONICET) in Argentina, and professor at La Plata National University (UNLP) in Argentina. She is interested in research assessment change and policies in Latin America, with a focus on research assessment ideas and instruments, incentives for open science and doctoral evaluation, and research training in higher education.

Laura Rovelli is a political scientist and has a PhD in Social Science from the University of Buenos Aires, Argentina. From 2020 to 2024, she coordinated the Latin American Forum for Research Assessment (FOLEC) from the Latin American Council of Social Sciences (CLACSO) and since 2022 onwards, she is a member of the Executive Board of DORA.

At present, she leads a research project on the potentialities and challenges in early academic career researchers in Latin America and is member of CLACSO’s WG “Open Science as a Common Good”.

Post navigation

Marta Wróblewska

(SWPS University, Poland)

Partner organisations

Australian Research Council (ARC)

An independent body reporting to the Australian Government Minister for Education and Youth. The ARC administers a significant component of Australia’s investment in research and development, provides advice to the Minister on matters related to research, and assesses the quality, engagement and impact of university research.

Dutch Research Council (NWO)

Dutch Research Council (NWO)

The national research council of the Netherlands, ensuring quality and innovation in science. NWO selects and funds research proposals based on the advice of experts from science and society from the Netherlands and abroad. NWO encourages national and international collaboration, invests in large-scale research facilities, promotes knowledge utilisation, and manages research institutes.

National Research Foundation of South Africa (NRF-SA)

National Research Foundation of South Africa (NRF-SA)

The primary public sector agency tasked with providing funding across the research system in South Africa. As the single largest funder of human capital development in the country, through its Research, Innovation and Impact Support and Advancement (RIISA) programme, the NRF offers funding opportunities along the entire research pipeline – from the next-generation researchers to emerging researchers and established researchers.

The Canadian Institutes of Health Research (CIHR)

Canadian Institutes of Health Research (CIHR)

Canada’s federal funding agency for health research. Composed of 13 Institutes, they collaborate with partners and researchers to support the discoveries and innovations that improve our health and strengthen the healthcare system.

ANVUR (Agenzia Nazionale Di Valutazione Del Sistema Universitario E Della Ricerca)

ANVUR, through its evaluation activities, aims to enhance the quality of higher education and research, with the objective of increasing its presence and visibility on the international stage, particularly within the European context. Through its initiatives, it contributes to ensuring the quality of the higher education system, promoting the achievement of increasingly high standards, guiding the allocation of public resources, and enhancing the international standing of institutions.  

Ministry of Education and Science of Ukraine

The Ministry of Education and Science of Ukraine’s goal is to build a sustainable system where everyone has equal opportunities for developing their talent throughout life.

MES creates and implements state policy to increase the nation’s education, develop science and technology, and ensure the EU standards and practices are being implemented in education in Ukraine.

Research Council Norway (RCN)

The Research Council of Norway (RCN) is the national funding agency for research and experimental development (R&D) and R&D-supported innovation of Norway. Our task is to make the best research and innovation possible. The Research Council works to foster new ideas, promote ground-breaking research and radical innovation and cultivate a society in which research is created, used and shared, and thus contributes to restructuring and enhanced sustainability.

Strand 1: Next-generation research assessment

Much interest – and in some instances hype – currently surrounds the ‘next generation’ of research metrics. Examples include new approaches for measuring novelty and disruptiveness, open science practices, societal impact, and methods that harness AI and emerging open data sources. In light of such developments, there is a strong case to revisit and update our understanding of metrics use for research assessment.

The fundamental research question underpinning this work is: Can research indicators and dashboards be developed for national assessment systems that harness promising advances in indicator methodologies and data, while also upholding principles of responsible metrics and responsible research assessment more broadly? This includes ensuring the diversity of research cultures is respected, voices of a diverse range of stakeholders are consulted, and unintended consequences are anticipated.

What are the possibilities and limitations for metric use in different kinds of assessment systems, with varying purposes and characteristics? For instance, how do considerations differ between assessments where funding and accountability are key purposes, versus assessments that produce strategic advice?

The Next Generation Research Assessment workstream tackles these timely questions through a mixed-methods, co-produced partnerships between research funders and meta-researchers. Crucially, we are placing our focus on the socio-technical, not just the technical dimensions of new indicators and data sources.

Strand 2: New geographies of research assessment

National systems for research assessment have gained increasing prominence in recent years. These systems are variously used to allocate institutional research funding, to gather intelligence about a country’s research strengths, and to induce behaviour change in the sector. These may pertain to enhancing research productivity or focusing research on particular thematic areas, but also often focus on tenets of RRA as expressed most notably through CoARA (Coalition for Advancing Research Assessment).

Besides a range of different aims, national research assessment systems also use a range of different methods and approaches and focus on different levels of assessment (e.g. institutions, department or individuals). As this diversity of systems increases, so does the need to systematically map them at a global level, to track evolution, convergence and divergence, emerging trends, and assess what works, and in what contexts.

Much of the literature on national research assessment focuses only on a small number of countries, typically from the global north. But to fully understand trends, convergences, divergences, and how different systems and contexts have different needs, a far more comprehensive perspective is required. 

The New Geographies of Research Assessment work strand addresses this need. It is a continuation of the AGORRA project, using the Atlas of Assessment – a free, open-access tool that maps research assessment systems worldwide. Launched in May 2025, the Atlas initially included 13 tester countries and continues to expand.

The New Geographies of Research Assessment work strand aims to populate the Atlas as far as possible. The ambition is for the Atlas to become a permanent tool with the fullest possible country coverage and regular updates to all countries included, contributing to a more complete and inclusive picture of research assessment systems.

The RoRI Atlas of Assessment
A platform offering expertly curated and quality-assured data on national research assessment systems from around the world

Strand 3: Indicators of research culture and environment (IRCE)

There is an increasing interest in monitoring and evaluating research cultures and environments as part of national, programme and institutional assessment processes. The RRA agenda aims to expand concepts of quality and excellence with the goal of creating incentives for improving research cultures and environments. This has led to a demand for relevant, trusted and reliable indicators that can provide evidence for monitoring and evaluation systems (the UK being just one recent example).

This strand of AGORRA provides a global assessment of the prospects for research culture indicators. This includes three components:

  • Providing a landscape view of qualities for which indicators would be valuable
  • A survey of indicators and forms of evidence in use
  • A practical review of implementation possibilities including a longitudinal evaluation of candidate indicators.

The workstream is led by the Curtin Open Knowledge Initiative in collaboration with CWTS and the core RORI team.

The workstream has identified the diversity of issues that are tackled globally under the heading of culture and will provide a critical analysis of the qualities of possible indicators. This will include advice on which indicators are appropriate for incorporation into responsible research assessment practices. It will explicitly identify indicators that are inappropriate for application within responsible research assessment systems as well as identifying if and how such indicators might be made fit for purpose over time.

Project timeline

AGORRA runs from early 2023 to mid 2028. There are several core planned milestones and outputs, as well as a range of follow-on options to ensure sustainability and optimal use of results.

AGORRA timeline, project plan & projected outputs


Transforming Assessment: the 2025 Global Research Council survey of funder approaches to responsible research assessment

Building on the influential 2020 report and subsequent action plans, the GRC’s 2025 global survey, developed in collaboration with RoRI, draws insights from 50 funding agencies to map the current landscape of RRA



Other related outputs from the AGORRA team include:


Related initiatives

CoARA (Coalition for Advancing Research Assessment) continues to gather support. The tally of organisational signatories to its underpinning Agreement on Reforming Research Assessment is rising by dozens each week and is now over 800. 

In May 2023, DORA (The Declaration on Research Assessment)—on whose foundations CoARA builds—marked its 10th anniversary with a series of workshops around the world. And at a national level, we have seen a sharper focus on these agendas in light of ongoing or proposed reforms to assessment frameworks in Australia, Czech Republic, Italy, New Zealand, Sweden and the UK.

A 2023 report on The Future of Research Evaluation by the InterAcademy Partnership, Global Young Academy and International Science Council, provides a guide to the waves of reform visible across assessment systems worldwide.

In China, there has been a renewed drive to break the influence of the “four-onlys” ( sì wéi or 四唯 in Mandarin) — shorthand for a reliance on “only papers, only titles, only education background, and only awards” to determine funding, hiring and promotion of researchers.  

This project builds on earlier contributions of RoRI team members to high profile initiatives in this area, such as the Leiden Manifesto, The Metric Tide and Next Generation Metrics. In 2020 RoRI partnered with DORA, UK Research and Innovation (UKRI) and the Global Research Council (GRC) to produce a working paper in support of the GRC Virtual Conference on Responsible Research Assessment

Besides those listed here, AGORRA also intends to support and collaborate with international networks and organisations at the forefront of these debates, including the International Science Council and its Centre for Science Futures, and ENRESSH (European Network for Research Evaluation in the Social Sciences and Humanities), which aims to improve evaluation procedures to take into account the diversity and wealth of SSH research.

Subscribe on Substack

We publish timely essays, reflections and signals from across metascience, research policy and reform. Expect original thinking, reactions to live debates, and practical insights from people studying research.