Matthew
Narratives
GRAIL
Funder Data Platform
AGORRA
Undisciplined
Peer Review
Portfolios
MetaROR
AFIRE
A series of talks and discussions for funders interested in formal experiments and evaluations of innovations in the research and innovation funding system, part of RoRI’s AFIRE programme: an Accelerator For Innovation & Research Funding Experimentation.
In partnership with the Innovation Growth Lab (IGL). Scroll to the bottom of the page for details on how to register.
18 December 2024 07:00 PST / 15:00 GMT / 16:00 CETRachel Kendal, Project Leader and Principal Investigator, CES Transformation Fund
The CES Transformation Fund is a grant scheme run by Rachel Kendal and team, on behalf of the Cultural Evolution Society (CES) thanks to the generosity of the John Templeton Foundation (Grant #61913: Transforming the field of cultural evolution and its application to global human futures).
The vision for the grant scheme was to boost, strengthen and extend scientific discoveries of cultural evolution. To achieve this the funding scheme explicitly aimed to encourage diversity through tackling (i) early career obstacles, (ii) western centrism, (iii) traditional disciplinary divides, and (iv) division of academics and policy makers.
These aims were embedded in the design of our application and review process which contained many measures to enhance equity diversity and inclusion (EDI) both in selecting who was funded and in the research conducted by our awardees.
This talk will provide an overview of the EDI measures we employed, including sequential anonymized-deanonymized review, a 2-stage application process incorporating mentoring, a diversity scoring system, an ‘ethics box’, and diverse review panels (see blog summary). The success of these measures in achieving our EDI objectives will be outlined as well as the practicalities of employing these measures. Survey responses from applicants and reviewers regarding the measures will also be discussed.
Prof Rachel Kendal received her PhD in Zoology from the University of Cambridge (UK) and has been based in the Anthropology Department at Durham University (UK) for most of her academic career. She developed an interest in EDI in funding through roles involving running funding competitions as (i) Director of Postgraduate Admissions for her department, (ii) founder of the Research Working Party for the Primate Society of Great Britian, and (iii) President of the Cultural Evolution Society when she applied for funding to run the CES Transformation Fund with scientific rigour and equity at its core.
19 February 2025 00:00 PST / 08:00 GMT/ 09:00 CETKen Emond, British Academy and Adrian Barnett, Queensland University of Technology
Adrian Barnett is a professor of statistics who has worked for over 29 years in health and medical research. He is interested in how we generate high quality scientific evidence that is of most value to the public and policy makers, and how this evidence gets translated into policy and practice. He has researched how research is funded, and has worked with funding agencies in Australia and overseas to improve their processes.
19 March 2025 08:00 PDT / 15:00 GMT / 16:00 CETMüge Simsek, Assistant Professor of Sociology, University of Amsterdam
Scientists and funding agencies invest considerable resources in writing and evaluating grant proposals. But do grant proposal texts noticeably change panel decisions in single blind review?
We report on a field experiment conducted by The Dutch Research Council (NWO) in collaboration with the authors in an early-career competition for awards of 800,000 euros of research funding. A random half of panelists were shown a CV and only a one-paragraph summary of the proposed research, while the other half were shown a CV and a full proposal. We find that withholding proposal texts from panelists did not detectibly impact their proposal rankings. This result suggests that the resources devoted to writing and evaluating grant proposals may not have their intended effect of facilitating the selection of the most promising science.
Müge Simsek is an Assistant Professor of Sociology in the programme group Institutions, Inequalities and Life Courses at the University of Amsterdam. She earned her PhD from Utrecht University in 2019 and completed postdoctoral research at both Utrecht University and the Netherlands Interdisciplinary Demographic Institute. Prior to her current role, she worked as a lecturer at University College Groningen. Her research centers on the integration processes of immigrants and their offspring, with a particular emphasis on the role of religion. In parallel, she maintains a secondary research agenda focused on the organization of science and inequality within academia.
23 April 2025 08:00 GMTHanna Denecke, Volkswagen Foundation
As it becomes increasingly difficult to find experts to carry out peer reviews, the Volkswagen Foundation in Hanover, Germany’s largest research funding organisation, is testing an alternative method: distributed peer review (DPR). In this approach, grant applicants review each other’s proposals. The process could make it easier to find suitable reviewers, especially as there is an incentive for reviewers to participate in order to have their own work considered. Applicants gain insights that could improve their own proposals by receiving more diverse feedback. DPR may also help to bring out more creative and daring research ideas.
While it’s not without challenges, including concerns about workload and potential competition, the early feedback from researchers is promising. Roughly 74% said they trusted the process to be fair in giving funding to the best research, and 70% of respondents said they thought it would help to identify more adventurous grant proposals than those selected by the existing peer review process, which is conducted by panelists appointed by the foundation.
Could this be a model for future grant processes in research funding? Further analysis seeks to answer this question.
Hanna Denecke is a passionate advocate for curiosity-driven research and the transformative power of bold, out-of-the-box ideas. As head of the Exploration team at the Volkswagen Foundation, she is committed to supporting groundbreaking research that has the potential to reshape the future of science and society. With a background in economics and extensive experience in research management, Hanna is keen to identify and promote experimental approaches to research funding and to further the Foundation’s mission of empowering researchers to explore bold, experimental paths that can ultimately transform both science and society.
All events are online and take place under the Chatham House rule. The meeting will provide an informal, high-trust environment in which the pros, cons and uncertainties of more experimental approaches to R&I funding can be shared and discussed among peers in the research and innovation funding community.
The meeting is by invitation only. Please register your interest by completing this brief form, or email Tom Stafford, AFIRE programme lead (t.stafford@sheffield.ac.uk), with any queries. A Zoom link and final agenda will be circulated in advance of each event.
23 October 2024 00:00 PDT / 08:00 BST / 09:00 CEST Anders Smith, Villum Foundation
Villum Foundation is a philanthropic foundation that supports technical and scientific research and education as well as environmental, social, and cultural projects in Denmark and internationally. In 2023, Villum Foundation allocated EUR 153 million in grants.
This talk provided an overview of the Villum Experiment grant programme, which targeted research proposals out of the ordinary that challenged the norm and had the potential to fundamentally change the way important topics are approached.
Evaluation is double-blind: the review panel only sees an anonymous research proposal; the panel doesn’t interact with each other, and the identity of the review panel is not disclosed.
The talk discussed the Foundation’s experiences with the programme and its reception among applicants and reviewers.
Anders Smith received his PhD in physics from the University of Copenhagen. He has worked in publishing, research and research administration at, among other places, Risø National Laboratory and the Technical University of Denmark. He has also served as a member of the Danish Board on Research Misconduct. For the past five years he has been Head of programme at the Danish philanthropic foundation Villum Foundation.
20 November 2024 07:00 PST / 15:00 GMT / 16:00 CETAlice Wu, Federation of American Scientists and Jordan Dworkin, Open Philanthropy
The conventional peer review process for reviewing grant proposals is known to be biased against riskier proposals. Expected utility — a formal quantitative measure of predicted success and impact — has been proposed by researchers Chiara Franzoni and Paula Stephan as a better metric for assessing the risk and reward profile of science proposals than traditional rubric evaluations.
Inspired by their paper, the Federation of American Scientists (FAS) developed a framework for implementing expected utility in peer review using forecasting, and conducted a pilot study of the proposed approach to test the mechanics and surface additional considerations for improvement.
Feedback from reviewers suggested that the process was user-friendly and the focus on just three key criteria—probability of success, scientific impact, and social impact—simplified the process, disentangled feasibility from impact, and eliminated biased metrics.
Future work on reducing ambiguities in the definition of project milestones and refining the impact scoring system will help make implicit project assumptions explicit and ensure that the desired information is solicited from reviewers.
Alice Wu is a Senior Associate at the Federation of American Scientists, specializing in clean energy policy. She has also contributed to FAS’s work on metascience policy—writing about science funding innovation and running a grant review pilot. Prior to joining FAS, Alice was a graduate student researcher at MIT. She received her SM in Electrical Engineering from MIT and her BS in Electrical Engineering from Columbia University.
Jordan Dworkin is the Innovation Policy Program Associate at Open Philanthropy, where he works to identify and support efforts to safely accelerate scientific progress and innovation. Previously, Jordan was the Metascience Program Lead at the Federation of American Scientists, and an Assistant Professor of Clinical Biostatistics in Psychiatry at Columbia University. He holds a PhD in Biostatistics from the University of Pennsylvania and a BS in Psychology from Haverford College.