May 11, 2026
Why generic indicators for research culture don’t work
New RoRI Working Paper explores the complexities of monitoring research culture and environment
While research culture and environment have long been priorities for research funders and policymakers, they remain among the most complex and contested areas of science policy. As global initiatives like CoARA and DORA accelerate the reform of research assessment, debate continues over whether (and how) indicators can be used to monitor these shifts.
A new RoRI Working Paper led by Cameron Neylon examines the feasibility of defining and tracking indicators for research culture and environment. The study is part of RoRI’s AGORRA (A Global Observatory of Responsible Research Assessment) project, which aims to accelerate the move toward more sophisticated, responsible research assessment.
What we looked at
The project followed three phases:
- Framework development: Examining how to categorise indicators in a way that respects different contexts and geographies.
- Information gathering: Mapping indicators currently in use, from traditional bibliographic data to institutional reporting.
- Feasibility analysis: Assessing the technical qualities and limitations of these indicators to see if they actually drive positive change.
The study explores how the qualities of research communities and organisations lead to the production of outputs, and how the resulting feedback is then received and integrated back into those same communities.
What we found
The research confirms that “research culture” and “environment” are not universal terms; their definitions vary significantly across systems and geographies. Consequently, there are no generic or universal indicators that can monitor research culture as a whole.
Key insights include:
- The Stability vs. Change model: In policy, “environment” usually refers to infrastructures where stability is needed, while “culture” refers to areas where change is desired.
- Structural data gaps: While we have a wealth of data on journal articles (particularly in STEM), there is a relative lack of information regarding the humanities, the health of research communities, and how organisations actually evolve in response to feedback.
- The feedback mismatch: Current evaluation systems (like periodic national assessments) are often backwards-looking and summative. These are ill-suited for culture change, which requires continuous, predictable feedback to aid learning.
Why this matters for funders and policymakers
Our findings suggest that while specific data can act as useful signposts for issues like research integrity or EDI, they should not be mistaken for a complete measurement of research culture.
Action points for funders and evaluation bodies:
- Be explicit: Clearly define whether your goal is gradual improvement, significant change, or consolidation before choosing an indicator.
- Avoid the “Metric Trap”: Plan for indicators to become less useful over time as communities adapt to them (perverse incentives).
- Shift to formative assessment: Move away from one-off burdensome assessments toward continuous monitoring that provides the stability necessary for productive change.
