Abstract
Research funding systems fundamentally influence how science operates. This paper aims to analyze the allocation of competitive research funding from different perspectives: How reliable are decision processes for funding? What are the economic costs of competitive funding? How does competition for funds affect doing risky research? How do competitive funding environments affect scientists themselves, and which ethical issues must be considered? We attempt to identify gaps in our knowledge of research funding systems; we propose recommendations for policymakers and funding agencies, including empirical experiments of decision processes and the collection of data on these processes. With our recommendations, we hope to contribute to developing improved ways of organizing research funding.
| Original language | English |
|---|---|
| Article number | e2407644121 |
| Pages (from-to) | 1-10 |
| Number of pages | 10 |
| Journal | Proceedings of the National Academy of Sciences of the United States of America |
| Volume | 121 |
| Issue number | 50 |
| Early online date | 2 Dec 2024 |
| DOIs | |
| Publication status | Published - 10 Dec 2024 |
Bibliographical note
Publisher Copyright:Copyright © 2024 the Author(s). Published by PNAS.
Funding
ACKNOWLEDGMENTS. S.C. received funding from the Fonds de la Recherche Scientifique (T.0177.21). G.S. thanks Stephan Pühringer, and U.S. thanks Ulf Heymann for valuable discussions. 2. J. P. A. Ioannidis, Fund people not projects. Nature 477, 529–531 (2011). 3. P. Kitcher, Science, Truth, and Democracy (Oxford University Press, 2001). 4. F. Shi, J. G. Foster, J. A. Evans, Weaving the fabric of science: Dynamic network models of science’s unfolding structure. Soc. Networks 43, 73–85 (2015). 5. R. K. Merton, The Sociology of Science: Theoretical and Empirical Investigations (University of Chicago Press, 1973). 6. S. Moore, C. Neylon, M. Paul Eve, D. Paul O’Donnell, D. Pattinson, “Excellence R Us”: University research and the fetishisation of excellence. Palgrave Commun. 3, 1–13 (2017). 7. T. Hellström, Homing in on excellence: Dimensions of appraisal in Center of Excellence program evaluations. Evaluation 17, 117–131 (2011). 8. P. O’Connor et al., Micro-political practices in higher education: A challenge to excellence as a rationalising myth? Crit. Stud. Educ. 61, 195–211 (2020). 9. D. Kahneman, O. Sibony, C. R. Sunstein, Noise: A Flaw in Human Judgment (Hachette UK, 2021). 10. J. P. Ioannidis, Z. Maniadis, In defense of quantitative metrics in researcher assessments. PLoS Biol. 21, e3002408 (2023). 11. D. Hicks, P. Wouters, L. Waltman, S. De Rijcke, I. Rafols, Bibliometrics: The Leiden Manifesto for research metrics. Nature 520, 429–431 (2015). 12. L. Leydesdorff, P. Wouters, L. Bornmann, Professional and citizen bibliometrics: Complementarities and ambivalences in the development and use of indicators—A state-of-the-art report. Scientometrics 109, 2129–2150 (2016). 13. U. Sandström, P. Van den Besselaar, Funding, evaluation, and the performance of national research systems. J. Informet. 12, 365–384 (2018). 14. O. Auranen, M. Nieminen, University research funding and publication performance—An international comparison. Res. Policy 39, 822–834 (2010). 15. M. Thelwall et al., Is research funding always beneficial? A cross-disciplinary analysis of U.K. research 2014–20. Quant. Sci. Stud. 4, 501–534 (2023). 16. J. McCullough, First comprehensive survey of NSF applicants focuses on their concerns about proposal review. Sci., Technol., Hum. Values 14, 78–88 (1989). 17. L. Bornmann, Scientific peer review. Ann. Rev. Inf. Sci. Technol. 45, 197–245 (2011). 18. L. Roumbanis, The oracles of science: On grant peer review and competitive funding. Soc. Sci. Inf. 60, 356–362 (2021). 19. U. Sandström, M. Hällsten, Persistent nepotism in peer-review. Scientometrics 74, 175–189 (2008). 20. V. Bakanic, C. McPhail, R. J. Simon, The manuscript review and decision-making process. Am. Sociol. Rev., 631–642 (1987). 21. T. Luukkonen, Conservatism and risk-taking in peer review: Emerging ERC practices. Res. Eval. 21, 48–60 (2012). 22. H. W. Marsh, L. Bornmann, R. Mutz, H. D. Daniel, A. O’Mara, Gender effects in the peer reviews of grant proposals: A comprehensive meta-analysis comparing traditional and multilevel approaches. Rev. Educ. Res. 79, 1290–1326 (2009). 23. Q. Wang, U. Sandström, Defining the role of cognitive distance in the peer review process with an explorative study of a grant scheme in infection biology. Res. Eval. 24, 271–281 (2015). 24. E. L. Pier et al., Low agreement among reviewers evaluating the same NIH grant applications. Proc. Natl. Acad. Sci. U.S.A. 115, 2952–2957 (2018). 25. R. Mutz, L. Bornmann, H. D. Daniel, Heterogeneity of inter-rater reliabilities of grant peer reviews and its determinants: A general estimating equations approach. PLoS ONE 7, e48509 (2012). 26. E. A. Erosheva, P. Martinková, C. J. Lee, When zero may not be zero: A cautionary note on the use of inter-rater reliability in evaluating grant peer review. J. R. Stat. Soc. Ser. A, Stat. Soc. 184, 904–919 (2021). 27. L. Bornmann, R. Mutz, H. D. Daniel, A reliability-generalization study of journal peer reviews: A multilevel meta-analysis of inter-rater reliability and its determinants. PLoS ONE 5, e14331 (2010). 28. J. O. Hesselberg, T. K. Dalsbo, H. Stromme, I. Svege, A. Fretheim, Reviewer training for improving grant and journal peer review. Cochrane Database Syst. Rev. 11, MR000056 (2023). 29. N. Graves, A. G. Barnett, P. Clarke, Funding grant proposals for scientific research: Retrospective analysis of scores by members of grant review panel. BMJ 343, d4797 (2011). 30. J. Coveney et al., ‘Are you siding with a personality or the grant proposal?’: Observations on how peer review panels function. Res. Integrity Peer Rev. 2, 19 (2017). 31. F. C. Fang, A. Bowen, A. Casadevall, NIH peer review percentile scores are poorly predictive of grant productivity. eLife 5, e13323 (2016). 32. N. S. Danthi, C. O. Wu, D. M. DiMichele, W. K. Hoots, M. S. Lauer, Citation impact of NHLBI R01 grants funded through the American Recovery and Reinvestment Act as compared to R01 grants funded through a standard payline. Circ. Res. 116, 784–788 (2015). 33. J. R. Kaltman et al., Prior publication productivity, grant percentile ranking, and topic-normalized citation impact of NHLBI cardiovascular R01 grants. Circ. Res. 115, 617–624 (2014). 34. S. A. Gallo et al., The validation of peer review through research impact measures and the implications for funding strategies. PLoS ONE 9, e106474 (2014). 35. M. S. Lauer, N. S. Danthi, J. Kaltman, C. Wu, Predicting productivity returns on investment: Thirty years of peer review, grant funding, and publication of highly cited papers at the National Heart, Lung, and Blood Institute. Circ. Res. 117, 239–243 (2015). 36. M. Reinhart, Peer review of grant applications in biology and medicine. Reliability, fairness, and validity. Scientometrics 81, 789–809 (2009). 37. S. A. Gallo, S. R. Glisson, External tests of peer review validity via impact measures. Front. Res. Metrics Anal. 3, 22 (2018). 38. L. Bornmann, L. Leydesdorff, P. Van den Besselaar, A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications. J. Informet. 4, 211–220 (2010). 39. O. Yaqub, D. Malkov, J. Siepel, How unpredictable is research impact? Evidence from the UK’s research excellence framework. Res. Eval. 32, 273–285 (2023). 40. S. Kahn, S. J. Ceci, W. M. Williams, “Is there gender bias in grant success? An extended meta-analysis. An extended meta-analysis” (Boston Univ. Questrom Sch. Bus. Res. Pap., 2023). 41. T. A. Hoppe et al., Topic choice contributes to the lower rate of NIH awards to African-American/black scientists. Sci. Adv. 5, eaaw7238 (2019). 42. P. S. Forscher, W. T. Cox, M. Brauer, P. G. Devine, Little race or gender bias in an experiment of initial review of NIH R01 grant proposals. Nat. Hum. Behav. 3, 257–264 (2019). 43. S. Gunashekar, S. Wooding, S. Guthrie, How do NIHR peer review panels use bibliometric information to support their decisions? Scientometrics 112, 1813–1835 (2017). 44. J. R. Kaltman et al., Prior publication productivity, grant percentile ranking, and topic-normalized citation impact of NHLBI cardiovascular R01 grants. Circ. Res. 115, 617–624 (2014). 45. I. Reymert, Bibliometrics in academic recruitment: A screening tool rather than a game changer. Minerva 59, 53–78 (2021). 46. M. H. MacRoberts, B. R. MacRoberts, Problems of citation analysis: A study of uncited and seldom-cited influences. J. Am. Soc. Inf. Sci. Technol. 61, 1–12 (2010). 47. A. Rushforth, S. De Rijcke, Practicing responsible research assessment: Qualitative study of faculty hiring, promotion, and tenure assessments in the United States. Res. Eval. 2024, rvae007 (2024). 48. G. Abramo, The forced battle between peer-review and scientometric research assessment: Why the CoARA initiative is unsound. Res. Eval. 2024, rvae021 (2024). 49. M. Dresler et al., Effective or predatory funding? Evaluating the hidden costs of grant applications. Immunol. Cell Biol. 101, 104–111 (2023). 50. A. Barnett, Funding schemes that cost as much as they reward (2021). https://medianwatch.netlify.app/post/hidden_funding_costs. Accessed 12 December 2023. 51. G. Schweiger, Can’t We Do Better? A cost-benefit analysis of proposal writing in a competitive funding environment PLoS ONE 18, e0282320 (2023). 52. T. von Hippel, C. von Hippel, To apply or not to apply: A survey analysis of grant writing costs and benefits. PLoS ONE 10, e0118494 (2015). 53. D. L. Herbert, A. G. Barnett, P. Clarke, N. Graves, On the time spent preparing grant proposals: An observational study of Australian researchers. BMJ Open 3, e002800 (2013). 54. A. A. Stone, C. A. Bachrach, J. B. Jobe, H. S. Kurtzman, V. S. Cain, The Science of Self-Report: Implications for Research and Practice (Psychology Press, 1999). 55. European-University-Association, “EFSI and horizon 2020: Efficiency and opportunity cost” (Tech. Rep., 2017). 56. S. Schneider, “Results of the 2018 FDP faculty workload survey. Technical report, the federal demonstration partnership” (Tech. Rep., 2020). 57. A. N. Link, C. A. Swann, B. Bozeman, A time allocation study of university faculty. Econ. Educ. Rev. 27, 363–374 (2008). 58. Research Councils UK, Report of the Research Councils UK Efficiency and Effectiveness of Peer Review Project (Research Councils UK, 2006).
| Funders | Funder number |
|---|---|
| Fonds De La Recherche Scientifique - FNRS | T.0177.21 |
| NHLBI | 617–624 |
Keywords
- competitive funding
- funding decision processes
- research funding
- science of science
Fingerprint
Dive into the research topics of 'The costs of competition in distributing scarce research funds'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver