Matthew

A study of cumulative advantages in funding allocation

Summary

The Matthew effect is one of the most widely discussed phenomena in research funding, and can be summed up as “the rich get richer and the poor get poorer.” Recent studies have highlighted complex dynamics at play in Matthew effects. We replicate two recent studies of Matthew effects using data from multiple funders. The first study (Bol et al., 2018) finds that researchers who have early success in winning funding have a greater chance of accumulating subsequent grants. The second study (Wang et al., 2019) suggests that applicants who are initially unsuccessful yet persist, reapply and then receive funding go on to outperform applicants who were successful on their first attempt. This has been dubbed the ‘what doesn’t kill you makes you stronger’ effect. Although both effects may plausible co-exist, they call for further research about their interrelation.

The MATTHEW project uses the same data to replicate the two studies. This will ensure a consistent comparison of the effects found in the two original studies. Secondly, our analysis may provide a unifying explanation of both findings, and will study the robustness and generalisability of both effects.

This project is a pilot of the capabilities of sharing data through RoRI’s Funder Data Platform (FDP).

The Matthew effect and early-career setbacks in research funding – a replication study (RoRI Working Paper No. 16)

CWTS-Leiden and Senior Research Fellow, RoRI

CFA-Aarhus and Research Fellow, RoRI

CFA-Aarhus and Senior Research Fellow, RoRI

Flavie Bidel

The Luxembourg National Research Fund (FNR)

Cindy Lopes Bento

The Luxembourg National Research Fund (FNR)

Philippe Vincent-Lamarre

Canadian Institutes of Health Research (CIHR)

Partner organisations

The project in detail

The Matthew effect is one of the most widely discussed phenomena in research funding. Merton (1968) proposed the Matthew effect to explain the accumulation of resources and recognition to the most successful, “the accruing of greater increments of recognition for particular scientific contributions to scientists of considerable repute and the withholding of such recognition from scientists who have not yet made their mark.”

In the years that have passed since then, interest in the phenomenon has not diminished, particularly due to signs of increasing concentration of funding among a small share of researchers alongside a growing share of early career researchers that work in temporary positions.

On the surface, the Matthew effect in research funding would seem to be determined by reviewers of applications and funders, reflecting to what extent weight is placed on status and past achievements when assessing applications. However, potential dynamics of funding and success may also depend on researchers, who may change their behaviour in reaction to funding decisions. Possible reactions for rejected applicants could be discouragement, though experiencing failure could also lead researchers to work even harder.

Similarly, grantees could strengthen their efforts due to increased self-confidence or alternatively become complacent.

Further evidence on these issues that examine dynamics across a wider range of funders could help to examine the validity of existing results and provide more robust answers to how or to what extent the Matthew effect operates.

Towards this aim, this study places focus on two recent studies.

Bol et al. (2019) finds that researchers who have early success in obtaining funding have a greater chance of accumulating subsequent grants and of continuing an academic career compared to researchers who narrowly miss out on funding. This calls into question the effectiveness of funding allocation and research culture, as many researchers with equally strong potential may not be able to continue to contribute their skills to the academic research sector if they do not win personal funding and build a sustainable research career.

However, applicants who experienced an early career setback and were unsuccessful in their early career grant application, but who reapplied to funding later on were shown to go on to outperform applicants who were successful on their first attempt (Wang et al., 2019) This has been interpreted as a “what doesn’t kill you makes you stronger” effect, i.e. the process of developing and then redeveloping a grant application informed by failure and funder feedback had strengthened their research program. Although these results may both be plausible and can co-exist they are somewhat paradoxical and call for further research.

We replicate both studies across seven research funders in Europe and North America (Canada, UK, Denmark, Luxembourg and Austria). The data we collect cover more than 80,000 funding applications and over a dozen different funding programs.

The use of identical data to replicate both studies provides a consistent basis for the effects compared to the two original studies, which involved different datasets and contexts, enabling the results to be more directly comparable. The inclusion of data from a suite of grant instruments from multiple funders provides a basis for the exploration of heterogeneous effects across multiple settings and may offer a unifying explanation of both studies.

The original methodology in the study by Bol et al. (2018) relied on a regression discontinuity design (RDD) to study the Matthew effect. In this approach, applicants who barely received funding (i.e. they were only slightly above the “funding line’”) were compared to applicants who barely missed receiving funding (i.e. they were only slightly below the “funding line”). The critical assumption here is that the two groups of applicants (near-hit and near-miss) are otherwise very comparable, and that it is near-random who gets selected for funding and who is rejected. With this assumption, any subsequent differences that we observe in terms of outcomes can be attributed causally to the funding decision.

The approach in the study by Wang et al. (2019), studying the “early career setback” is very similar, but suffers from one particular problem: it is not looking at the entire population of scholars who experienced an early career setback (i.e. getting a first grant application rejection), but only at the subset of scholars who reapply to another funding round at a later point in time. In order to address this problem, they use various methods: (1) a RDD regression approach with controls; (2) a matching approach; and (3) a “conservative removal” approach. We will focus on the third approach in particular.

In the data collected in the Matthew project, there is no hard cut-off based on review scores in most funding programs, making it difficult to implement a regression discontinuity design. Instead, we implement a fuzzy regression discontinuity design. Where the ordinary regression discontinuity design focuses on the local effects around the funding threshold only, we aim to study the generalisability of the effects based on a hierarchical Bayesian model using latent variables.

Project timeline

The MATTHEW project runs from April 2023 until June 2025.

Initiation phase: April 2023 – December 2023

Prepare collaboration agreements to facilitate data sharing.
Further revise and refine methodology and planning, based on discussion with partners.
Prepare a data template with details of all requested data elements.

Data phase: January – September 2024

Deposition of requested data from partners on the Funder Data Platform.
Matching of applicants to authors in Dimensions.

Analysis phase: September 2024 – March 2025

Replicate the Matthew effect study.
Replicate the early-career setback study.
Integrate findings of two replication studies.

Writing phase: March 2025 – June 2025

Writing report of results
Present results internally to participating partners and RoRI more broadly.
Prepare academic publication on the topic.

Outputs

The Matthew project expects to result in a RoRI Working Paper which is intended to be published in the academic literature. In addition to being presented in international venues, it will also be presented internally to RoRI, with a particular focus on policy recommendations.

Main menu