Research on Research

Using Natural Language Processing (NLP) in classification of Social Sciences and Humanities research and societal impact
2024-11-11 13:00 - 14:30 UTC
The European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH) in partnership with the Research on Research Institute (RoRI) is proud to present the next webinar in its series on research evaluation as it is practiced across disciplines and countries.

We are opening a new thematic line on AI in research assessment with the presentation of two classification exercises using Natural Language Processing (NLP).

MODERATORSpecial advisor Dr Jon Holm, Research Council of Norway

SPEAKERSTopic modeling of SSH publications from the VABB publication databaseSenior researcher Dr Raf Guns, Flemish Centre for Research & Development Monitoring (ECOOM), Antwerp University

Classifying REF impact cases by user groups – a lexical approachProfessor Andrea Bonaccorsi, Economics and Management at the School of Engineering of the University of Pisa

RESPONDENTDr Denis Newman-Griffis Lecturer in Data Science, University of Sheffield, and a Research Fellow of the Research on Research Institute

SPEAKERS

Raf Guns (he) is a senior researcher at the University of Antwerp, where he coordinates the Antwerp branch of the Flemish Centre for Research & Development Monitoring (ECOOM). His research focuses on quantitative science studies, addressing topics like interdisciplinarity, open science, and the characteristics of the social sciences and humanities.

Andrea Bonaccorsi (he) is a Professor of Economics and Management at the School of Engineering of the University of Pisa. His main areas of research are economics of science and innovation. With more than 200 works, he is listed in the top 2% of world scientists.

Denis Newman-Griffis(they/them) is a Lecturer in Data Science at the University of Sheffield, a Research Fellow of the Research on Research Institute, and a British Academy Innovation Fellow. Their research investigates the principles and practices of Responsible AI, focusing on health and disability as well as public sector governance. They lead the Research on Research Institute’s GRAIL project on Responsible AI and Machine Learning for research funding and evaluation, and they are an active participant in Responsible AI policy discussions in research, education, and government.

Register on Eventbrite.

Introducing MetaROR: A new peer review platform for metaresearch
2024-11-26 15:00 - 16:30
The Research on Research Institute (RoRI) and the Association for Interdisciplinary Meta-Research and Open Science (AIMOS) are launching the MetaROR (MetaResearch Open Review) platform.

MetaROR will facilitate open peer review of articles in the field of metaresearch, and will operate according to a publish-review-curate model. This model will accelerate the communication of scholarly work since peer review will take place quickly, openly, and after publication rather than before.

In this online seminar, Kathy Zeiler and Ludo Waltman, the Editors-in-Chief of MetaROR, will introduce the platform and discuss the benefits of MetaROR’s innovative publish-review-curate model.

In addition, authors of articles reviewed by MetaROR will share their experiences, and journal editors will reflect on how they expect to collaborate with MetaROR to streamline publication processes.

We invite anyone interested in metaresearch or innovation in scientific publishing to join this seminar. We look forward to an engaging event!

AGENDA

I. What is MetaROR?Kathy Zeiler and Ludo Waltman, the Editors-in-Chief of MetaROR

II. Why do authors submit to MetaROR?Short perspectives from authors of two articles submitted to MetaROR and Q&ASimon Porter and Natalia Gonzalez Bohorquez

III. How does MetaROR enrich the landscape of scientific publishing?Short perspectives from the editors of two metaresearch journals and an editor of eLife , with Q&ALin Zhang, Wolfgang Kaltenbrunner, and Peter Rodgers

SPEAKERS

Ludo WaltmanLudo Waltman is professor of Quantitative Science Studies at Leiden University’s Centre for Science and Technology Studies (CWTS) and co-chair of the Research on Research Institute (RoRI). He serves as Editor-in-Chief of MetaROR.

Kathy ZeilerKathy Zeiler is Nancy Barton Scholar and Professor of Law at Boston University’s School of Law. She serves as a board member of the Association for Interdisciplinary Meta-Research and Open Science (AIMOS) and as Editor-in-Chief of MetaROR.

Natalia Gonzalez BohorquezNatalia Gonzalez Bohorquez (MPH, BS Soc) is an early career researcher in health economics and health services research. She recently finished her doctoral research (PhD) focused on exploring liveable communities for people with disabilities living in rural areas of Australia.

Lin ZhangLin Zhang is professor at the School of Information Management at Wuhan University. She serves as Editor-in-Chief of Scientometrics.

Wolfgang KaltenbrunnerWolfgang Kaltenbrunner is senior researcher at Leiden University’s Centre for Science and Technology Studies and senior research fellow at the Research on Research Institute (RoRI). He serves as associate editor of Science as Culture.

Peter RodgersPeter Rodgers was part of the team that launched eLife in 2012, and is currently Chief Magazine Editor of the journal. Previously he was Chief Editor of Nature Nanotechnology and Editor of Physics World magazine.

Simon PorterSimon Porter, VP of Research Futures at Digital Science, has spent a career exploring research metadata with particular focus on the ways metadata moves through systems, as well as its use in creating digital research identity.

Utility of LLMs in identifying and assessing academic genres
2024-12-03 13:00 - 14:30
The European Network for Research Evaluation in the Social Sciences and the Humanities (ENRESSH) in partnership with the Research on Research Institute (RoRI) is proud to present the next webinar in its series on research evaluation as it is practiced across disciplines and countries.

This is the second event in a thematic line on AI in research assessment. This time we ask how large language models (LLMs) can support assessment of various academic genres.

INTRODUCTIONResearch assessment as intertextual reading: Opportunities and challenges in the use of Artificial intelligence in evaluation of SSHSpecial advisor Dr Jon Holm, Research Council of Norway

SPEAKERSEvaluating social science, arts and humanities journal article quality with ChatGPTProfessor Mike Thelwall, University of Sheffield

Research quality evaluation of journal articles is time-consuming, even for post-publication expert review tasks like national research evaluation exercises. It is also necessary to assess the strength of candidates’ works for recruitment and promotion. This talk assesses ChatGPT’s ability to score social science, arts and humanities journal articles using papers and criteria from the UK’s Research Excellence Framework 2021. The systemic implications of using artificial intelligence to fully or partially replace human judgement for this core task will also be discussed.

Adaptation of neural language models to different textual domains and the use of language models for comparative analysis of textDr Denis Newman-Griffis, Senior Lecturer, University of Sheffield, and a Research Fellow of the Research on Research Institute

Speakers

Mike Thelwall (he) is a Professor of Data Science in the Information School at the University of Sheffield in the UK. He primarily investigates quantitative methods to support research evaluation, including artificial intelligence, citation analysis and altmetrics. He has recently shown that ChatGPT can provide useful research quality assessments for published journal articles. His books include: Quantitative Methods in Research Evaluation Citation Indicators, Altmetrics, and Artificial Intelligence (https://doi.org/10.48550/arXiv.2407.00135). He is an associate editor of the Journal of the Association for Information Science and Technology and sits on five other editorial boards.

Denis Newman-Griffis (they/them) is a Senior Lecturer at the University of Sheffield Centre for Machine Intelligence and a British Academy Innovation Fellow. They lead the Research on Research Institute’s GRAIL project on Responsible AI and Machine Learning for research funding and evaluation, and they are an active participant in Responsible AI policy discussions in research, education, and government.

Jon Holm (he) is a special advisor at the Research council of Norway where he works with the development of national research assessment in Norway and the use of AI in research evaluation and analysis. Jon and Denis have recently published on the potential and pitfalls of the use of AI in research financing: Holm et al. (2025). “Big Data for Big Investments: Making Responsible and Effective Use of Data Science and AI in Research Councils” in Nielsen et al (ed.) Artificial Intelligence and Evaluation, Routledge 2025 (ch. 7)

Do Grant Proposal Texts Matter for Funding Decisions? A Field Experiment
2025-03-19 15:00 - 16:00
19 March 2025 08:00 PDT / 15:00 GMT / 16:00 CET

Scientists and funding agencies invest considerable resources in writing and evaluating grant proposals. But do grant proposal texts noticeably change panel decisions in single blind review?

We report on a field experiment conducted by The Dutch Research Council (NWO) in collaboration with the authors in an early-career competition for awards of 800,000 euros of research funding. A random half of panelists were shown a CV and only a one-paragraph summary of the proposed research, while the other half were shown a CV and a full proposal. We find that withholding proposal texts from panelists did not detectibly impact their proposal rankings. This result suggests that the resources devoted to writing and evaluating grant proposals may not have their intended effect of facilitating the selection of the most promising science.

About the Speaker

Müge Simsek is an Assistant Professor of Sociology in the programme group Institutions, Inequalities and Life Courses at the University of Amsterdam. She earned her PhD from Utrecht University in 2019 and completed postdoctoral research at both Utrecht University and the Netherlands Interdisciplinary Demographic Institute. Prior to her current role, she worked as a lecturer at University College Groningen. Her research centers on the integration processes of immigrants and their offspring, with a particular emphasis on the role of religion. In parallel, she maintains a secondary research agenda focused on the organization of science and inequality within academia.

An experiment with Distributed Peer Review
2025-04-23 08:00 - 09:00
As it becomes increasingly difficult to find experts to carry out peer reviews, the Volkswagen Foundation in Hanover, Germany’s largest research funding organisation, is testing an alternative method: distributed peer review (DPR). In this approach, grant applicants review each other’s proposals. The process could make it easier to find suitable reviewers, especially as there is an incentive for reviewers to participate in order to have their own work considered. Applicants gain insights that could improve their own proposals by receiving more diverse feedback. DPR may also help to bring out more creative and daring research ideas.

While it’s not without challenges, including concerns about workload and potential competition, the early feedback from researchers is promising. Roughly 74% said they trusted the process to be fair in giving funding to the best research, and 70% of respondents said they thought it would help to identify more adventurous grant proposals than those selected by the existing peer review process, which is conducted by panelists appointed by the foundation.

Could this be a model for future grant processes in research funding? Further analysis seeks to answer this question.

Hanna Denecke is a passionate advocate for curiosity-driven research and the transformative power of bold, out-of-the-box ideas. As head of the Exploration team at the Volkswagen Foundation, she is committed to supporting groundbreaking research that has the potential to reshape the future of science and society. With a background in economics and extensive experience in research management, Hanna is keen to identify and promote experimental approaches to research funding and to further the Foundation’s mission of empowering researchers to explore bold, experimental paths that can ultimately transform both science and society.﻿

Do we need a new generation of public institutions for ST&I?
2025-04-30 16:00 - 18:00
Do we need a new generation of public institutions for ST&I?

Public institutions for science and technology have hardly changed over half a century; even the UK’s most recent creation, ARIA, is consciously modelled on a US institution founded in the 1960s. But there is much to learn from other sectors that have innovated far more, including business. The deep institutional conservatism of public science may be one factor behind the continuous decline of R&D productivity.

This talk from Sir Geoff Mulgan, Professor of Collective Intelligence, Public Policy and Social Innovation, UCL STEaPP & Co-founder, The Institutional Architecture Lab (TIAL), will provide a diagnosis, share alternative ways of thinking being used by some governments around the world, and point to a potentially very different institutional landscape for science and technology in the 2030s.

This event will be chaired by James Wilsdon, Professor of Research Policy, UCL & Executive Director, Research on Research Institute.

Anjana Ahuja, Science Commentator at the Financial Times, William Cullerne Bown, Journalist at The Independent and New Scientist, and Ine Steenmans, Associate Professor in Futures, Analysis and Policy at UCL, will join as discussants.

Article in ResearchProfessional covering the seminar

Flashpoints & faultlines in science, technology, engineering & public policy: a STEaPP seminar series

Science, technology and engineering policy has never been more important, nor more contested. Over the next year, UCL’s Department of Science, Technology, Engineering and Public Policy (STEaPP) will be holding a series of seminars under the umbrella theme of ‘Flashpoints & faultlines in science, technology, engineering and public policy’. These are designed to provoke and stimulate fresh lines of thought, debate and research within the department, across UCL and beyond.

All staff and students in STEaPP are warmly invited to participate, as are colleagues from across UCL with an interest in these topics. The series is being convened by Geoff Mulgan and James Wilsdon, and is being organised with the support of the Research on Research Institute (RoRI), based within STEaPP.

Two-stage versus continuous application processing
2025-05-21 16:00 - 17:00
Curious about how different grant processes shape outcomes for both applicants and funders? In this talk, we share our experiences with continuous and two-stage processes – from shrinking average waiting times by more than 60 % through continuous handling, to reducing overall writing effort by nearly half using a two-stage model.

Our analyses reveal how each approach influences submission patterns, applicant satisfaction, and administrative load. We will present key insights from our evaluations (including comparisons with other funders), highlighting when and why a particular process may be most effective.

More and More and More: why energy transitions are an unhelpful myth
2025-05-27 16:00 - 18:00
Think we’re transitioning to green energy? Think again. Join us for a talk on the history, mythology and politics of energy transitions.

Join us for a compelling discussion with Jean-Baptiste Fressoz, historian and researcher at CNRS & the Alexandre Koyré Center of EHESS in Paris, as he forensically dissects the past and future potential for energy transitions.

Fressoz’s new book More and More and More; an all-consuming history of energy – described by The Economist as ‘necessary, eye-opening and frequently gobsmacking’ – challenges the widely held assumption that our energy history is a linear progression from wood to coal, oil, nuclear, and ultimately to green energy. Instead, he argues that each new phase of energy use remains inextricably linked to its predecessors, creating a complex web of interdependencies.

This seminar, the second in our new series on ‘Flashpoints & faultlines’ will be chaired by Neil Morisetti (Professor of Climate and Resource Security, and Head of the Department for Science, Technology, Engineering and Public Policy (STEaPP) at UCL, and former UK Government Special Representative for Climate Change).

We will also have contributions from discussants: Dr Bipashyee Ghosh, Lecturer in Engineering, Innovation and Public Policy at STEaPP-UCL, and Oliver Morton (Senior and Briefings Editor, The Economist).

Flashpoints & faultlines in science, technology, engineering & public policy: a STEaPP seminar series

Science, technology and engineering policy has never been more important, nor more contested. Over the next year, UCL’s Department of Science, Technology, Engineering and Public Policy (STEaPP) will be holding a series of seminars under the umbrella theme of ‘Flashpoints & faultlines in science, technology, engineering and public policy’. These are designed to provoke and stimulate fresh lines of thought, debate and research within the department, across UCL and beyond.

All staff and students in STEaPP are warmly invited to participate, as are colleagues from across UCL with an interest in these topics. The series is being convened by Geoff Mulgan and James Wilsdon, and is being organised with the support of the Research on Research Institute (RoRI), based within STEaPP.

Seminars will take place from 4pm to 5:30pm, on the last Tuesday or Wednesday of the month, and will be followed by an informal drinks and nibbles reception to 6pm. The seminars are held in person as an opportunity to strengthen collaborative links across STEaPP and wider UCL.

Launching Funding by Algorithm: A Handbook for research funders on using AI
2025-06-20 12:00 - 13:00
Join us for the launch of Funding by Algorithm: A handbook for responsible uses of AI and machine learning by research funders – a major new publication from the Research on Research Institute (RoRI).

AI is already transforming research systems around the world. But how can funders make informed, ethical, and effective choices about using AI in their work?

For the last two years, RoRI’s GRAIL project has worked closely with research funders from around the world to understand how funders can responsibly and effectively make use of AI and machine learning technologies in the vital work of research funding and assessment.

Our new handbook Funding by Algorithm distils insights, strategies, and case studies from this global collaboration into a practical, easy-to-use handbook for funders and organisations navigating the fast-changing world of AI in research systems.

At this online launch event, we’ll introduce the handbook, share key lessons from the GRAIL project, and hear reflections from leading research funders already engaging with these technologies. Whether you’re cautious, curious, or already experimenting with AI, this event will offer valuable guidance on how to move forward responsibly.

You will hear from speakers including:

James Wilsdon, Executive Director of the Research on Research Institute

Denis Newman-Griffis, University of Sheffield and GRAIL Project Lead

Katrin Milzow, Swiss National Science Foundation

Jon Holm, Research Council of Norway

Inés Bouzón, “la Caixa” Foundation

Alexandra Apavaloae, Social Science and Humanities Research Council of Canada

Jennifer Gold, ESRC and UK Metascience Unit

Lidia Borrell-Damian, Science Europe

Metascience 2025 Conference
2025-06-30 08:00 - 2025-07-07 17:00
Save the Date:

June 30 – July 2, 2025University College London

From 30 June to 2 July 2025, the Research on Research Institute (RoRI) and the Center for Open Science (COS) welcomed over 800 researchers, funders, policymakers, and innovators from 65 countries to explore the future of science and research systems.

Metascience 2025 marked one of RoRI’s biggest milestones to date and the largest gathering of the global metascience community. Attendees came from across research, funding, publishing, policy, technology, and related sectors to share evidence, exchange ideas, and reflect on how research systems can be improved.

Metascience 2025 showcased the power of collaboration across the metascience community, leaving a lasting legacy of knowledge, connections, and inspiration for the future of research.

Conference sessions

Watch the main sessions from Metascience 2025, covering policymaking, funding, the science of science, AI, new institutions, and open knowledge infrastructures

See the full story of Metascience 2025 on our Wakelet, featuring photos, media coverage, and highlights from the conference.

ECR Research on Research Seminar Series (ECR RoRSS)
2026-03-01 08:00 - 2026-12-31 17:00
We’re launching a new series of online talks by early career researchers (ECRs) on metascience / research on research topics (in their broadest senses). Each talk could be focused on completed research you want to share, work in progress you want to brainstorm with colleagues, or ideas for advancing metascience or future research you’d like to try out.

Anyone self-identifying as an ECR can present, and seminar attendance will be open to all ( we strongly encourage senior colleagues to join!). Webinars will be held on a monthly basis at lunchtimes (UK/EU) throughout the first half of 2026, with flexible timing to include ECRs from across the globe.

We see a lot of potential in this, and we hope this series becomes a vibrant, inclusive, and valuable space for knowledge-sharing and network-building for ECRs and the wider metascience community.

If you’d like to present, suggest a speaker, or stay updated on upcoming talks, please fill in this form or email us!

Email contact: a.l.butters@sheffield.ac.uk

ECR RoRSS organising committee

Anna Butters (Sheffield/RoRI), Becky Ioppolo (Cambridge), Mollie Etheridge (Anglia Ruskin/Cambridge), Allison Beggs (Cambridge), Melanie Benson Marshall (Sheffield/RoRI), Josie Coburn (UCL/RoRI), Youyou Wu (UCL/RoRI), Similo Ngwenya (UCL/RoRI), Seunghyun Lee (UCL/RoRI)
