What are you looking for?
Ask me a question

GRAIL

Getting responsible about AI and machine learning in research funding and evaluation

Summary

Artificial intelligence (AI) is transforming research systems around the world. Research funders have a central role in enabling new AI innovations, but they are also increasingly AI users: as global research funding and assessment become more data-driven, funders are exploring the use of AI and machine learning to better leverage their deep knowledge and extensive data about the research sector.

A UK funder study of potential uses of AI in national research evaluation sparked debate in the journal Nature.

The GRAIL project is developing the new knowledge, evidence, and practical guidance to help ensure that research funders are equipped to use AI effectively, ethically, and equitably in research funding and assessment. GRAIL has three core workstreams:

  • Building shared knowledge through a discussion-based workshop series on AI use cases and practicalities in funding and assessment;
  • Understanding current practice in how funders are exploring and applying AI in different contexts;
  • Shaping future practice by producing resources for shared understanding and practical steps for making responsible use of AI and machine learning.

Project team

Denis Newman-Griffis

Project lead, University of Sheffield

Project lead, RoRI/UCL

Helen Buckley Woods

Senior Research Fellow

hbw@researchonresearch.org

Helen joined RoRI in 2019, and in 2023 moved as part of RoRI’s core team to the Department of Science, Technology, Engineering and Public Policy (STEaPP) at University College London (UCL). She is now a Senior Research Fellow in Metascience, and leads or co-leads a number of RoRI’s flagship projects.

Helen is a social scientist with a doctorate in higher education studies. Her thesis: Knowledge production and disciplinary practices in a British University: A qualitative cross-disciplinary case study explored researchers’ views about knowledge production, evaluation, and dissemination in British higher education. 

Before joining RoRI, Helen worked for ten years at Sheffield’s School of Health and Related Research (ScHARR) as an information scientist. At ScHARR she led information retrieval and data management activities for systematic reviews and evidence syntheses projects, commissioned by organisations such as the National Institute for Health and Care Excellence (NICE), Department of Health and Social Care, and the Joseph Rowntree Foundation. 

Projects
RoRi publications
Woods HB and Pinfield S. Incentivising research data sharing: a scoping review. Wellcome Open Research 2022, 6:355 Innovating peer review, reconfiguring scholarly communication Kaltenbrunner, W., Pinfield, S., Waltman, L., Woods HB and Brumberg, J. (2022). Innovating peer review, reconfiguring scholarly communication: an analytical overview of ongoing peer review innovation activities. Journal of Documentation. Vol. 78 No. 7, pp.429-449 The experimental research funder’s handbook Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives (RoRI Working Paper No.9) Experiments with randomisation in research funding: scoping and workshop report (RoRI Working Paper No.4) Why draw lots? Funder motivations for using partial randomisation to allocate research grants (RoRI Working Paper No.7) GRAIL: Codesigning responsible uses of AI in research funding and evaluation(RoRI Working Paper No. 13) UNDISCIPLINED: How do research funders define transdisciplinary research? (RoRI Working Paper No. 12) Taming complexity: narrative CVs in grant funding evaluations (RoRI Working Paper No. 14) Fast and flexible research funding: a review (RoRI Working Paper No. 20) Funding by Algorithm – A handbook for responsible uses of AI and machine learning by research funders AI and Reviewer Matching in Research Funding: Three Case Studies

Researcher, UCL

Youyou Wu

UKRI AI Metascience Early Career Fellow

UKRI AI Metascience Early Career Fellow, RoRI and Lecturer, University College London

Youyou Wu is an Associate Professor of Psychology at University College London, based in the Department of Psychology and Human Development at IOE.

Trained as a psychologist, her research applies computational methods, including machine learning and natural language processing, to the study of individual differences such as personality. More recently, her work has expanded into metascience, focusing on how AI can improve research systems.

She joined the Research on Research Institute (RoRI) in 2024 as part of the GRAIL project, developing best practices for the use of AI in research funding. She also contributes to the AFIRE project, which helps funders design AI-related experiments.

In 2025, she was awarded the UKRI Metascience AI Early Career Fellowship to study the influence of generative AI on the language of science. 

Youyou’s project will investigate whether generative AI is reinventing the language of science in research publications, and the implications for peer review. It builds on her previous work at RoRI, where she developed AI guidance for research funders worldwide. She will collaborate with the UK Metascience Unit to translate insights from her research into policy recommendations. The fellowship will connect her with international metascience researchers, including a parallel cohort in the US and Canada, creating opportunities to collaborate and contribute to global debates on AI and the future of research.

Projects
RoRi publications
Funding by Algorithm – A handbook for responsible uses of AI and machine learning by research funders Funding by Algorithm: Insights and Recommendations

Mike Thelwall

Researcher, University of Sheffield

Partner organisations

Australian Research Council (ARC)

An independent body reporting to the Australian Government Minister for Education and Youth. The ARC administers a significant component of Australia’s investment in research and development, provides advice to the Minister on matters related to research, and assesses the quality, engagement and impact of university research.

Board representatives
Claire Forsyth
Ke Zhang
Projects
Austrian Science Fund (FWF)

Austrian Science Fund (FWF)

Funds and supports outstanding researchers and their teams in their work at the cutting edge of research. They are given the freedom they need to conduct independent research and take unexpected directions. The international peer review process ensures quality in the selection of the best researchers and ideas from all disciplines.

Board representative
Falk Reckling
Projects
Dutch Research Council (NWO)

Dutch Research Council (NWO)

The national research council of the Netherlands, ensuring quality and innovation in science. NWO selects and funds research proposals based on the advice of experts from science and society from the Netherlands and abroad. NWO encourages national and international collaboration, invests in large-scale research facilities, promotes knowledge utilisation, and manages research institutes.

Board representatives
Jurgen Ganzevles
Puck Wildschut
Projects
"la Caixa" Foundation

la Caixa Foundation

The major philanthropic institution in Spain and, with an annual budget of 600 million euros in 2024, one of the biggest in Europe. Its mission is to contribute to building a better and fairer society, giving more opportunities to those most in need, with the values of trust, excellence and social commitment. With 120 years of existence, the “la Caixa” Foundation main areas of action are social welfare, education, research and innovation and culture.

Board representatives
Carla Carbonell Cortés
Patricia Nadal
Projects
Novo Nordisk Foundation

Novo Nordisk Foundation

An independent Danish foundation with corporate interests. Their aims are to provide a stable basis for the commercial and research activities of the companies in the Novo Group (Novo Nordisk A/S and Novozymes A/S (Novonesis), and to support scientific, humanitarian and social causes.

Board representative
Gert Vilhelm Balling
Projects
UK Research and Innovation

UK Research and Innovation

A non-departmental public body of the Government of the United Kingdom that directs research and innovation funding.

Projects
Social Sciences and Humanities Research Council (SSHRC)

Social Sciences and Humanities Research Council (SSHRC)

A Canadian federal research funding agency that promotes and supports research and training in the humanities and social sciences.

Board representatives
Matthew Lucas
Dominique Roche
Projects
Swiss National Science Foundation (SNSF)

Swiss National Science Foundation (SNSF)


The leading Swiss organisation for the promotion of scientific research. In close collaboration with higher education institutions and other partners, the SNSF works towards creating the best possible conditions for the development and international integration of Swiss research.

Board representatives
Katrin Milzow
Srdjan Perko
Projects
Volkswagen Foundation

Volkswagen Foundation

The largest German private non-profit organisation involved in the promotion and support of academic research. 

Board representatives
Johanna Brumberg
Antje Tepperwien
Projects
Wellcome

Wellcome

A global charitable foundation supporting science to solve the urgent health issues facing everyone. They work with policy makers, run advocacy campaigns, and form partnerships with other organisations to ensure everyone benefits from advances in health science. 

Board representatives
Alyson Fox
Kirsten Bound
Projects

Research Council Norway (RCN)

The Research Council of Norway (RCN) is the national funding agency for research and experimental development (R&D) and R&D-supported innovation of Norway. Our task is to make the best research and innovation possible. The Research Council works to foster new ideas, promote ground-breaking research and radical innovation and cultivate a society in which research is created, used and shared, and thus contributes to restructuring and enhanced sustainability.

Projects

The project in detail

In 2021, RoRI and the Research Council of Norway co-hosted a series of three virtual workshops on the use of AI and machine learning (AI/ML) technologies in research funding and evaluation.

Those workshops, summarised in a 2022 joint RoRI/RCN Working Paper, highlighted the need for developing clear, practical shared understanding of the potential roles of AI/ML technologies for research funders, and how funders could go about using these tools effectively, ethically, and equitably.

The GRAIL project responds to this need with three in-depth work streams:

Strand 1: Building shared knowledge

Research funders around the globe have been experimenting with AI and machine learning in their own organisations, but funders often lack opportunities to learn from each other, share successes, and solve common problems. 

To close this gap, the GRAIL project has been built around a cross-funder workshop series including twelve focused discussions over two years. GRAIL workshops are coproductive spaces in which funders come together to discuss a particular use case for AI/ML or how to tackle a specific practical challenge in using technologyAI/ML within their own organisations.

GRAIL workshops offer a much-needed space for research funders to exchange knowledge and experiences with AI, while building a shared base of practice to support new applications.

Strand 2: Understanding current practice

To better support future uses of AI/ML in research funding and assessment, GRAIL is conducting the first data collection on current AI/ML applications by funders.

In partnership with RoRI’s AGORRA project, we collected data on AI/ML applications in research assessment from funders around the world, as part of the Global Research Council 2025 survey on responsible research assessment. Our findings, including organisational perspectives on managing AI/ML processes in practice, are published in the 2025 GRC report.

We are also working with our global consortium of funders in GRAIL to collect detailed examples of AI/ML use cases in research funding processes, illustrating the diverse purposes AI/ML are being put to and the challenges involved in responsible use.

Strand 3: Shaping future practice

The GRAIL project aims to ensure that future applications of AI/ML technologies by research funders are as well-informed as possible and build on a shared base of good practice. To facilitate this, we are producing a handbook on responsible use of AI and machine learning for research funders, as a go-to reference for funders and others in research systems.

Funding by Algorithm is launching in June 2025, and addresses a wide range of key knowledge for AI/ML use in funding contexts:

  • What funders need to know about AI/ML methods
  • Policy contexts driving AI/ML adoption in funding
  • Key steps involved in any AI/ML application
  • Organisational challenges and strategies for using AI/ML
  • Real-world case studies of AI/ML use by GRAIL partners
  • Recommendations for best practice with AI/ML for funders

Project timeline

GRAIL runs from April 2023 – June 2025.

Strand 1

The GRAIL workshop series included twelve workshops between June 2023 – April 2025:

  • June 2023: ChatGPT/Generative AI and the research funding ecosystem
  • November 2023: AI and research evaluation
  • January 2024: GRAIL & AI guidance
  • February 2024: Natural language processing in research funding
  • April 2024: Policy and responsible use of AI/ML
  • June 2024: Applying AI to improve research assessment
  • July 2024: Guidelines for the use of generative AI in research funding processes
  • September 2024: Responsible AI principles for research funders
  • November 2024: Human in the Loop
  • February 2025: Collaboration and reuse: tools, data, and knowledge structures
  • March 2025: Competencies and collaboration on AI/ML applications
  • April 2025: Impact assessment, documentation and reporting, and transparency and reliability

Strand 2

Data collection on AI in research assessment:

  • GRAIL collaborated with the AGORRA project to design survey questions on the use of AI/ML technologies in responsible research assessment in Winter-Spring 2024.
  • The survey was administered by AGORRA during Summer 2024 – Winter 2025.
  • RoRI’s report on the GRC survey will be published May 2025.

Data collection on AI in research funding: internal to funders participating in GRAIL; our survey has run from September 2024 – May 2025.

Strand 3

Funding by Algorithm was developed September 2024 – April 2025, and will be published June 2025.

Outputs

Responsible AI handbook

Funding by Algorithm

A handbook for responsible uses of AI and machine learning by research funders

Funding by Algorithm was launched as a diamond open-access publication by RoRI in June 2025. The handbook includes the following sections:

  • Part 1: Foundations for AI/ML
  • Part 2: The case and context for AI/ML
  • Part 3: Practical guide to applying AI/ML
  • Part 4: Organisational perspectives and collaboration
  • Part 5: Case studies
  • Part 6: Responsible AI futures

AI and Reviewer Matching in Research Funding

Three case studies exploring how funders are using AI to match grant applications with reviewers.

Reports

Publications

Presentations

Funding by Algorithm: Insights and Recommendations

Key lessons, guidance and directions from the GRAIL project

Subscribe on Substack

We publish timely essays, reflections and signals from across metascience, research policy and reform. Expect original thinking, reactions to live debates, and practical insights from people studying research.