Machine learning, metrics & merit: the future of research assessment

The use of quantitative indicators and metrics in research assessment continues to generate a mix of enthusiasm, hostility and critique. To these possibilities, we can add growing interest in uses of machine learning and artificial intelligence (AI) to automate assessment processes, and reduce the cost and bureaucracy of conventional methods of peer and panel-based review.

Novel methods also bring potential pitfalls, uncertainties and dilemmas, and may operate in some tension with moves towards responsible research assessment, as reflected in the Declaration on Research Assessment (DORA) and the new Coalition for Advancing Research Assessment (CoARA).

As the UK again reviews its approach to research assessment and the design of the Research Excellence Framework (REF), these and other issues are up for discussion through the Future Research Assessment Programme (FRAP), initiated by the four UK higher education funding bodies.

This workshop launches two new studies that should make significant contributions to the FRAP process.

The first, led by Professor Mike Thelwall, is a ground-breaking analysis of whether one could run a REF exercise using AI. The second is an updated review of the role of metrics in the UK research assessment system, which builds on the 2015 review,The Metric Tide, which called for responsible approaches to the use of metrics, and cautioned against purely metric-based approaches to assessment. For more on these studies, see recent articles in Nature, Research Professional and Times Higher Education.

We were joined by Professor Dame Jessica Corner, new Executive Chair of Research England who offered opening keynote remarks, and by two panels of UK and international experts.