Rethinking research assessment at the OECD

Above: Lin Zhang (Wuhan University/ RoRI’s AGORRA project) speaking at the meeting

On 27-28 January, the RoRI team visited Paris for a workshop hosted by the OECD’s Global Science Forum (GSF) on rethinking research assessment and incentives. The meeting brought together researchers, funders and policymakers to reflect on progress in assessment reform globally, and the role that the OECD itself could play. 

RoRI’s researchers and partners were well represented in the agenda and audience, with contributions from Alex Rushforth (CWTS/RoRI); Lin Zhang (Wuhan University/ AGORRA project), Moumita Kouley (IISC/AGORRA project); Cameron Neylon (AGORRA project); Jon Holm (Research Council of Norway); and Steven Hill (UKRI). 

A few impressions stayed with us from the meeting. One was the need to see indicators and rankings not simply as tools of measurement, but as governance and market mechanisms that shape institutional behaviour. 

A session on university rankings was especially insightful, with speakers such as Jelena Brankovic interrogating the ongoing “datafying” of university management. We also heard examples of good practice, such as Sorbonne University, which as of this year has chosen to no longer supply data to the Times Higher and other university rankings illustrating that universities can also assert agency in these processes.

In his concluding remarks, which opened the final session of the meeting, James Wilsdon, RoRI’s Executive Director, warned against the dangers of creating a circular discussion among those who are already converted to responsible assessment. This is particularly important when these agendas appear to be slipping down the list of political priorities – for example, in the European Commission’s high-level goals for Framework Programme 10.

Renewed processes of political mobilisation are required to push these issues back up the agenda – underpinned by robust and systematic metascientific evidence. A lot of these discussions remain very normative, and we need better evidence to show how current research systems are underperforming because goals, incentives and evaluation methods are poorly aligned. 

James Wilsdon also pointed to the influence of OECD’s own STI indicators in embedding certain proxies for quality and impact in the way we benchmark and compare the strength and performance of different R&D systems – albeit at a more aggregated level. This sends powerful signals back through those systems as to what matters and merits being counted. So potentially one of the OECD’s biggest impacts here could be in ensuring that its own indicators are selected and applied in ways that reinforce RRA through the global research system.

Alongside the assessment workshop, we held an informal roundtable with 10-12  colleagues in the OECD’s science, technology and innovation division to discuss the changing landscape for metascience. We also held meetings with Claire Giry, President, and her colleagues at the French National Research Agency (ANR), the policy team at the International Science Council (ISC); and PSL University to explore their potential involvement in one or two new RoRI projects. 

Overall, the trip highlighted how assessment reform is now at the centre of how research systems adapt to new societal expectations. Both at the OECD, and in the French research system, we found lively debate and a palpable sense that the international metascience community is coalescing around shared questions, even if the answers are still emerging.