Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Alexandru Marcoci; David P. Wilkinson; Ans Vercammen; Bonnie C. Wintle; Anna Lou Abatayo; Ernest Baskin; Henk Berkman; Erin M. Buchanan; Sara Capitán; Tabaré Capitán; Ginny Chan; Kent Jason G. Cheng; Tom Coupé; Sarah Dryhurst; Jianhua Duan; John E. Edlund; Timothy M. Errington; Anna Fedor; Fiona Fidler; James G. Field; Nicholas Fox; Hannah Fraser; Alexandra L. J. Freeman; Anca Hanea; Felix Holzmeister; Sanghyun Hong; Raquel Huggins; Nick Huntington-Klein; Magnus Johannesson; Angela M. Jones; Hansika Kapoor; John Kerr; Melissa Kline Struhl; Marta Kołczyńska; Yang Liu; Zachary Loomas; Brianna Luis; Esteban Méndez; Olivia Miske; Fallon Mody; Carolin Nast; Brian A. Nosek; E. Simon Parsons; Thomas Pfeiffer; W. Robert Reed; Jon Roozenbeek; Alexa R. Schlyfestone; Claudia R. Schneider; Andrew Soh; Zhongchen Song; Anirudh Tagat; Melba Tutor; Andrew H. Tyner; Karolina Urbanska; Sander van der Linden

doi:10.1038/s41562-024-01961-1

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Alexandru Marcoci
, David P. Wilkinson
, Ans Vercammen
, Bonnie C. Wintle
, Anna Lou Abatayo
, Ernest Baskin
, Henk Berkman
, Erin M. Buchanan
, Sara Capitán
, Tabaré Capitán
, Ginny Chan
, Kent Jason G. Cheng
, Tom Coupé
, Sarah Dryhurst
, Jianhua Duan
, John E. Edlund
, Timothy M. Errington
, Anna Fedor
, Fiona Fidler
, James G. Field

Nicholas Fox, Hannah Fraser, Alexandra L. J. Freeman, Anca Hanea, Felix Holzmeister, Sanghyun Hong, Raquel Huggins, Nick Huntington-Klein, Magnus Johannesson, Angela M. Jones, Hansika Kapoor, John Kerr, Melissa Kline Struhl, Marta Kołczyńska, Yang Liu, Zachary Loomas, Brianna Luis, Esteban Méndez, Olivia Miske, Fallon Mody, Carolin Nast, Brian A. Nosek, E. Simon Parsons, Thomas Pfeiffer, W. Robert Reed, Jon Roozenbeek, Alexa R. Schlyfestone, Claudia R. Schneider, Andrew Soh, Zhongchen Song, Anirudh Tagat, Melba Tutor, Andrew H. Tyner, Karolina Urbanska, Sander van der Linden

Research output: Contribution to Journal › Article › Academic › peer-review

Abstract

Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

Original language	English
Pages (from-to)	287-304
Journal	Nature Human Behaviour
Volume	9
Issue number	2
DOIs	https://doi.org/10.1038/s41562-024-01961-1
Publication status	Published - 1 Feb 2025
Externally published	Yes

Funding

This research was developed with funding from the Defense Advanced Research Projects Agency (DARPA) under cooperative agreement nos. HR001118S0047 (A.M., H.F., A.H., A.V., F.M., B.C.W., F.F., D.P.W.), HR00112020015 and N660011924015 (A.L.A., M.K.S., N.F., O.M., Z.L., E.S.P., A.H.T., B.L., T.M.E., B.A.N.) and N66001-19-C-4014 (T.P., F.H., M.J., Y.L.). Following the DARPA model, the funder played a role in conceptualization and design, but was not involved in data collection, analysis, decision to publish or preparation of the manuscript. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the US Government. T.M.E., N.F., Z.L., B.L., O.M., B.N. and A.H.T. thank M. v. Assen, R. v. Aert, M. Bakker and M. Sitnikov for help with power calculations and calculation of effect sizes; V. Ashok for contributions to the replication outcome data; and the hundreds of researchers who conducted the replications, served as editors and reviewers for the preregistration review process, and helped conduct power analyses and additional statistical consulting. A.M. thanks D. Siegel for research assistance, and the audience of the SKAPE seminar at the University of Edinburgh for helpful feedback on an earlier version of this paper.

Funders	Funder number
U.S. Department of Defense
US government
Defense Advanced Research Projects Agency	N660011924015, HR001118S0047, N66001-19-C-4014, HR00112020015

Access to Document

10.1038/s41562-024-01961-1

Persistent URL (handle)

Persistent link

Cite this

Marcoci, A., Wilkinson, D. P., Vercammen, A., Wintle, B. C., Abatayo, A. L., Baskin, E., Berkman, H., Buchanan, E. M., Capitán, S., Capitán, T., Chan, G., Cheng, K. J. G., Coupé, T., Dryhurst, S., Duan, J., Edlund, J. E., Errington, T. M., Fedor, A., Fidler, F., ... van der Linden, S. (2025). Predicting the replicability of social and behavioural science claims in COVID-19 preprints. Nature Human Behaviour, 9(2), 287-304. https://doi.org/10.1038/s41562-024-01961-1

@article{a45e8d9fc3db4bc597db4b0724646b24,

title = "Predicting the replicability of social and behavioural science claims in COVID-19 preprints",

abstract = "Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise ({\textquoteleft}beginners{\textquoteright}) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise ({\textquoteleft}experienced{\textquoteright}). For experienced individuals, the average accuracy was 0.57 (95\% CI: [0.53, 0.61]) after interaction, and they correctly classified 61\% of claims; beginners{\textquoteright} average accuracy was 0.58 (95\% CI: [0.54, 0.62]), correctly classifying 69\% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of {\textquoteleft}fast science{\textquoteright} under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.",

author = "Alexandru Marcoci and Wilkinson, \{David P.\} and Ans Vercammen and Wintle, \{Bonnie C.\} and Abatayo, \{Anna Lou\} and Ernest Baskin and Henk Berkman and Buchanan, \{Erin M.\} and Sara Capit{\'a}n and Tabar{\'e} Capit{\'a}n and Ginny Chan and Cheng, \{Kent Jason G.\} and Tom Coup{\'e} and Sarah Dryhurst and Jianhua Duan and Edlund, \{John E.\} and Errington, \{Timothy M.\} and Anna Fedor and Fiona Fidler and Field, \{James G.\} and Nicholas Fox and Hannah Fraser and Freeman, \{Alexandra L. J.\} and Anca Hanea and Felix Holzmeister and Sanghyun Hong and Raquel Huggins and Nick Huntington-Klein and Magnus Johannesson and Jones, \{Angela M.\} and Hansika Kapoor and John Kerr and \{Kline Struhl\}, Melissa and Marta Ko{\l}czy{\'n}ska and Yang Liu and Zachary Loomas and Brianna Luis and Esteban M{\'e}ndez and Olivia Miske and Fallon Mody and Carolin Nast and Nosek, \{Brian A.\} and \{Simon Parsons\}, E. and Thomas Pfeiffer and Reed, \{W. Robert\} and Jon Roozenbeek and Schlyfestone, \{Alexa R.\} and Schneider, \{Claudia R.\} and Andrew Soh and Zhongchen Song and Anirudh Tagat and Melba Tutor and Tyner, \{Andrew H.\} and Karolina Urbanska and \{van der Linden\}, Sander",

year = "2025",

month = feb,

day = "1",

doi = "10.1038/s41562-024-01961-1",

language = "English",

volume = "9",

pages = "287--304",

journal = "Nature Human Behaviour",

issn = "2397-3374",

publisher = "Nature Publishing Group",

number = "2",

}

Marcoci, A, Wilkinson, DP, Vercammen, A, Wintle, BC, Abatayo, AL, Baskin, E, Berkman, H, Buchanan, EM, Capitán, S, Capitán, T, Chan, G, Cheng, KJG, Coupé, T, Dryhurst, S, Duan, J, Edlund, JE, Errington, TM, Fedor, A, Fidler, F, Field, JG, Fox, N, Fraser, H, Freeman, ALJ, Hanea, A, Holzmeister, F, Hong, S, Huggins, R, Huntington-Klein, N, Johannesson, M, Jones, AM, Kapoor, H, Kerr, J, Kline Struhl, M, Kołczyńska, M, Liu, Y, Loomas, Z, Luis, B, Méndez, E, Miske, O, Mody, F, Nast, C, Nosek, BA, Simon Parsons, E, Pfeiffer, T, Reed, WR, Roozenbeek, J, Schlyfestone, AR, Schneider, CR, Soh, A, Song, Z, Tagat, A, Tutor, M, Tyner, AH, Urbanska, K & van der Linden, S 2025, 'Predicting the replicability of social and behavioural science claims in COVID-19 preprints', Nature Human Behaviour, vol. 9, no. 2, pp. 287-304. https://doi.org/10.1038/s41562-024-01961-1

TY - JOUR

T1 - Predicting the replicability of social and behavioural science claims in COVID-19 preprints

AU - Marcoci, Alexandru

AU - Wilkinson, David P.

AU - Vercammen, Ans

AU - Wintle, Bonnie C.

AU - Abatayo, Anna Lou

AU - Baskin, Ernest

AU - Berkman, Henk

AU - Buchanan, Erin M.

AU - Capitán, Sara

AU - Capitán, Tabaré

AU - Chan, Ginny

AU - Cheng, Kent Jason G.

AU - Coupé, Tom

AU - Dryhurst, Sarah

AU - Duan, Jianhua

AU - Edlund, John E.

AU - Errington, Timothy M.

AU - Fedor, Anna

AU - Fidler, Fiona

AU - Field, James G.

AU - Fox, Nicholas

AU - Fraser, Hannah

AU - Freeman, Alexandra L. J.

AU - Hanea, Anca

AU - Holzmeister, Felix

AU - Hong, Sanghyun

AU - Huggins, Raquel

AU - Huntington-Klein, Nick

AU - Johannesson, Magnus

AU - Jones, Angela M.

AU - Kapoor, Hansika

AU - Kerr, John

AU - Kline Struhl, Melissa

AU - Kołczyńska, Marta

AU - Liu, Yang

AU - Loomas, Zachary

AU - Luis, Brianna

AU - Méndez, Esteban

AU - Miske, Olivia

AU - Mody, Fallon

AU - Nast, Carolin

AU - Nosek, Brian A.

AU - Simon Parsons, E.

AU - Pfeiffer, Thomas

AU - Reed, W. Robert

AU - Roozenbeek, Jon

AU - Schlyfestone, Alexa R.

AU - Schneider, Claudia R.

AU - Soh, Andrew

AU - Song, Zhongchen

AU - Tagat, Anirudh

AU - Tutor, Melba

AU - Tyner, Andrew H.

AU - Urbanska, Karolina

AU - van der Linden, Sander

PY - 2025/2/1

Y1 - 2025/2/1

N2 - Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

AB - Replications are important for assessing the reliability of published findings. However, they are costly, and it is infeasible to replicate everything. Accurate, fast, lower-cost alternatives such as eliciting predictions could accelerate assessment for rapid policy implementation in a crisis and help guide a more efficient allocation of scarce replication resources. We elicited judgements from participants on 100 claims from preprints about an emerging area of research (COVID-19 pandemic) using an interactive structured elicitation protocol, and we conducted 29 new high-powered replications. After interacting with their peers, participant groups with lower task expertise (‘beginners’) updated their estimates and confidence in their judgements significantly more than groups with greater task expertise (‘experienced’). For experienced individuals, the average accuracy was 0.57 (95% CI: [0.53, 0.61]) after interaction, and they correctly classified 61% of claims; beginners’ average accuracy was 0.58 (95% CI: [0.54, 0.62]), correctly classifying 69% of claims. The difference in accuracy between groups was not statistically significant and their judgements on the full set of claims were correlated (r(98) = 0.48, P < 0.001). These results suggest that both beginners and more-experienced participants using a structured process have some ability to make better-than-chance predictions about the reliability of ‘fast science’ under conditions of high uncertainty. However, given the importance of such assessments for making evidence-based critical decisions in a crisis, more research is required to understand who the right experts in forecasting replicability are and how their judgements ought to be elicited.

UR - http://www.scopus.com/inward/record.url?scp=85212492299&partnerID=8YFLogxK

U2 - 10.1038/s41562-024-01961-1

DO - 10.1038/s41562-024-01961-1

M3 - Article

SN - 2397-3374

VL - 9

SP - 287

EP - 304

JO - Nature Human Behaviour

JF - Nature Human Behaviour

IS - 2

ER -

Predicting the replicability of social and behavioural science claims in COVID-19 preprints

Abstract

Funding

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this