Undisciplined
Future models of funding and evaluating transdisciplinary research [completed]
Filter
What are you looking for?
Type
Tags
Sort
Portfolios
Research funding landscape analysis
Peer Review
RoRI Atlas of Peer Review
PATHWAYS
Advancing Research on Research Use to Enhance Positive Impact
Narratives
The uses and evaluation of researchers’ narrative CVs
MetaROR
Translating Publish-Review-Curate outputs into actionable signals for research funders: A MetaROR Study
Matthew
A study of cumulative advantages in funding allocation [completed]
GRAIL
Getting responsible about AI and machine learning in research funding and evaluation [completed]
Funder Data Platform
A Secure Collaboration Hub for Metascience
Criteria
Exploring gender inequality in research funding
Atlas of Assessment
New geographies of research assessment
Narratives
The uses and evaluation of researchers’ narrative CVs
Summary
As reforms in science assessment gain momentum, an increasing number of funding agencies and academic institutions are adopting Narrative CVs. These alternative formats reduce the emphasis on quantifiable metrics and instead incorporate narrative sections that allow applicants to describe their accomplishments in a more reflective, discursive manner. The introduction of Narrative CVs is driven by the aim to foster greater diversity within the scientific community and broaden the scope of recognised scientific contributions. Despite these aspirations, there is limited empirical evidence on how Narrative CVs affect research evaluation in practice. This project investigates their influence by examining how they are used in peer review processes for competitive research funding. It aims to answer the following questions:
- How does use of Narrative CVs impact the reliance on more established criteria of quality in research, and how does such use change over time?
- What practical and cultural factors shape the evaluative use of Narrative CVs?
- How can funders facilitate impactful uses of Narrative CVs?
Project team
CWTS-Leiden and Senior Research Fellow, RoRI
Senior Research Fellow, RoRI and Senior Researcher, Centre for Science and Technology Studies (CWTS), Leiden University
Wolfgang Kaltenbrunner is a senior researcher at the Centre for Science and Technology Studies at Leiden University in the Netherlands. His research focuses on the politics of quantification in evaluation processes; academic curricula vitae as devices for comparison in peer review; as well as practices of publishing, reviewing, and editing academic literature.
Wolfgang joined the RoRI team in 2022 and co-leads RoRI’s project on narrative CVs.
Projects
RoRi publications
Implementing Narrative CVs – Five recommendations for Funders
Taming complexity: narrative CVs in grant funding evaluations (RoRI Working Paper No. 14)
Peer review in funding organizations: An analytical literature review (RoRI Working Paper No.11)
Kaltenbrunner, W., Pinfield, S., Waltman, L., Woods HB and Brumberg, J. (2022). Innovating peer review, reconfiguring scholarly communication: an analytical overview of ongoing peer review innovation activities. Journal of Documentation. Vol. 78 No. 7, pp.429-449
Waltman, L., Kaltenbrunner, W., Pinfield, S., & Woods, H. B. (2023). How to improve scientific peer review: Four schools of thought. Learned Publishing.
Woods HB, Brumberg J, Kaltenbrunner, W et al. An overview of innovations in the external peer review of journal manuscripts. Wellcome Open Research 2023, 7: 82
Executive Director of RoRI
Executive Director, RoRI and Professor of Research Policy at University College London (UCL)
James is one of the founders of RoRI and has been its director since 2019. He is also Professor of Research Policy at University College London (UCL), based in its Department of Science, Technology, Engineering & Public Policy (STEaPP). Since the late-1990s, as a researcher, writer, adviser and campaigner, James has worked at the heart of science and research policy in the UK and internationally. In addition to academic posts at the universities of Sheffield, Sussex and Lancaster, he has worked in think tanks, NGOs and as director of science policy for the Royal Society, the UK’s national academy of sciences.
In his work, James has advanced concepts such as upstream public engagement, science diplomacy, responsible metrics and responsible research assessment; and he has co-founded or chaired initiatives such as People & Planet; the Campaign for Social Science; International Network for Government Science Advice (INGSA); UK Forum for Responsible Research Metrics; and the Research on Research Institute (RoRI).
In 2014, James was asked to chair an independent UK government review of research metrics, published as The Metric Tide. He subsequently chaired a European Commission expert group on Next Generation Metrics, and in 2022, revisited these debates with colleagues in Harnessing the Metric Tide.
In 2015, he was elected a Fellow of the UK’s Academy of Social Sciences, and in 2022, of the International Science Council.
Projects
RoRi publications
Harnessing the Metric Tide: indicators, infrastructures & priorities for UK responsible research assessment
Stafford T, Rombach I, Hind D et al. Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives. Wellcome Open Research 2023, 8:309
Hook, D. W. & Wilsdon, J. R. (2023). The pandemic veneer: COVID-19 research as a mobilisation of collective intelligence by the global research community. Collective Intelligence, 2(1).
UNDISCIPLINED: How do research funders define transdisciplinary research? (RoRI Working Paper No. 12)
Fast and flexible research funding: a review (RoRI Working Paper No. 20)
A new typology of national research assessment systems: continuity and change in 13 countries (RoRI Working Paper No.15)
Why draw lots? Funder motivations for using partial randomisation to allocate research grants (RoRI Working Paper No.7)
Where next for partial randomisation of research funding? The feasibility of RCTs and alternatives (RoRI Working Paper No.9)
Good practice in the use of machine learning & AI by research funding organisations (RoRI Working Paper No. 10)
Research Fellow, RoRI
Research Fellow, RoRI and Researcher, Centre for Science and Technology Studies (CWTS), Leiden University
Judit Varga is a Researcher at the Centre for Science and Technology Studies (CWTS) at Leiden University in the Netherlands. She is a social scientist with a doctorate in science and technology studies (STS) (University of Nottingham, 2021) and a background in psychology. Her work focuses on exploring the interactions between quantitative and interpretative methods in social science; the politics of quantification; values in science (evaluation); interdisciplinarity; and science in society.
Her doctoral research provided new methodological insights about combining STS and scientometrics, and explored interdisciplinary interactions in a subfield of computational social science.
As part of RoRI, Judit studies how narrative CVs shape science evaluation with Helen Buckley Woods and Wolfgang Kaltenbrunner.
Michaela Strinzel
Swiss National Science Foundation (SNSF)
Partner organisations
DORA is a global initiative that campaigns for and supports improving ways to assess research and researchers.
DORA started as the San Francisco Declaration on Research Assessment. The idea to write the declaration was developed in 2012 during at the Annual Meeting of the American Society for Cell Biology in San Francisco. Since 2018, it has become a worldwide initiative covering all scholarly disciplines and all key actors in the global scholarly system including researchers, funders, publishers, academies and professional societies, and research performing institutions.
Projects
The Luxembourg National Research Fund (FNR) is the main funder of research activities in Luxembourg. We invest public funds and private donations into research projects in various branches of science and the humanities, with an emphasis on selected core strategic areas. Furthermore, we support and coordinate activities to strengthen the link between science and society and to raise awareness for research. We also advise the Luxembourg government on research policy and strategy.
Projects
An integrated health research organisation created from the consolidation of Michael Smith Foundation for Health Research (MSFHR) and British Columbia Academic Health Science Network (BC AHSN) to support the growth and evolution of BC’s vibrant health research system and life sciences sector.
Board representatives
Ellie Griffith
Kevin Sauve
Projects
The national research council of the Netherlands, ensuring quality and innovation in science. NWO selects and funds research proposals based on the advice of experts from science and society from the Netherlands and abroad. NWO encourages national and international collaboration, invests in large-scale research facilities, promotes knowledge utilisation, and manages research institutes.
Board representatives
Jurgen Ganzevles
Puck Wildschut
Projects
Board representatives
Srdjan Perko
Projects
A non-departmental public body of the Government of the United Kingdom that directs research and innovation funding.
The largest German private non-profit organisation involved in the promotion and support of academic research.
Board representatives
Johanna Brumberg
Antje Tepperwien
Projects
A global charitable foundation supporting science to solve the urgent health issues facing everyone. They work with policy makers, run advocacy campaigns, and form partnerships with other organisations to ensure everyone benefits from advances in health science.
Board representatives
Alyson Fox
Kirsten Bound
Projects
The project in detail
We explore how Narrative CVs are used across three different funding programs, each located in distinct national contexts and targeting applicants at distinct career phases: the Rubicon grant of the Dutch NWO aimed at applicants within 1 year from their PhDs, who wish to conduct a 1-2 year-long research project abroad; the Ambizione grant of the SNSF for scholars within 1-4 years of their PhD, who wish to set up their first individual project; and finally the Momentum grant of VW, aimed at tenured professors who aim to develop a new research line.
Within each funding program, we also study two distinct disciplinary panels: one from the social sciences and humanities (SSH) and one from science, technology, engineering and mathematics (STEM), yielding a total of 6 panels. Finally, for each panel, we study the use of Narrative CVs in two subsequent application rounds, yielding a total of 12 instances of panel observations.
Methods
We conduct participant observation during panel meetings, making detailed fieldnotes about how panellists discussed applicants’ achievements. In addition, we conduct (online) semi-structured interviews with a selection of panellists after each observation to better understand personal views and experiences with Narrative CVs. We thematically analyse interview transcripts and fieldnotes, focusing on instances when evaluators use Narrative CVs.
Preliminary findings
We have identified key features of the evaluative environment in which Narrative CVs are used, providing leverage points for funders to facilitate the desirable use of Narrative CVs. We highlight environmental features along the following dimensions:
- Key actors (such as the composition of review panels and choice of panel chairs)
- Key practices (such as the practical interpretation of evaluative criteria by reviewers, or also the way in which funders and chairs handle resistance to the introduction of Narrative CVS)
- The mode in which reviewers engage with the inevitable contingency of research assessment (namely, in terms of formal procedures, or rather in terms of values and situated decisions).
Project timeline
The NARRATIVES project runs for two years, to July 2025.
Outputs
This RoRI Insights report outlines five key recommendations to help funders create a supportive environment for the effective implementation of Narrative CVs.
- Varga, J. & Kaltenbrunner, W. (2024). Taming complexity: narrative CVs in grant funding evaluations. RoRI working paper 14. DOI: 10.6084/m9.figshare.27291537. Available on Figshare.
- Varga, J. & Kaltenbrunner, W. (Nov 5, 2024). Narrative CVs: How do they change evaluation practices in peer review for grant funding? Blogpost.
- Varga, J. & Kaltenbrunner, W. (Sep 19, 2024). Taming complexity: narrative CVs in grant funding evaluations. Presentation at STI Berlin 2024, Sep 18-20.
In this Working Paper, we propose a way to conceptualize how narrative CVs alter evaluative practices in peer review and provide preliminary findings about their impact from an ongoing study. We draw on observations and interviews with reviewers in two subsequent funding rounds of a Dutch Research Council (NWO) grant programme, which aims to enable early career researchers in the social sciences and humanities to carry out an independent research project abroad.
