Subjective evidence evaluation survey for many-analysts studies

Alexandra Sarafoglou; Suzanne Hoogeveen; Don Van Den Bergh; Balazs Aczel; Casper J. Albers; Tim Althoff; Rotem Botvinik-Nezer; Niko A. Busch; Andrea M. Cataldo; Berna Devezer; Noah N.N. Van Dongen; Anna Dreber; Eiko I. Fried; Rink Hoekstra; Sabine Hoffman; Felix Holzmeister; Jürgen Huber; Nick Huntington-Klein; John Ioannidis; Magnus Johannesson; Michael Kirchler; Eric Loken; Jan Francois Mangin; Dora Matzke; Albert J. Menkveld; Gustav Nilsonne; Don Van Ravenzwaaij; Martin Schweinsberg; Hannah Schulz-Kuempel; David R. Shanks; Daniel J. Simons; Barbara A. Spellman; Andrea H. Stoevenbelt; Barnabas Szaszi; Darinka Trübutschek; Francis Tuerlinckx; Eric L. Uhlmann; Wolf Vanpaemel; Jelte Wicherts; Eric Jan Wagenmakers

doi:10.1098/rsos.240125

Subjective evidence evaluation survey for many-analysts studies

Alexandra Sarafoglou^*, Suzanne Hoogeveen, Don Van Den Bergh, Balazs Aczel, Casper J. Albers, Tim Althoff, Rotem Botvinik-Nezer, Niko A. Busch, Andrea M. Cataldo, Berna Devezer, Noah N.N. Van Dongen, Anna Dreber, Eiko I. Fried, Rink Hoekstra, Sabine Hoffman, Felix Holzmeister, Jürgen Huber, Nick Huntington-Klein, John Ioannidis, Magnus JohannessonMichael Kirchler, Eric Loken, Jan Francois Mangin, Dora Matzke, Albert J. Menkveld, Gustav Nilsonne, Don Van Ravenzwaaij, Martin Schweinsberg, Hannah Schulz-Kuempel, David R. Shanks, Daniel J. Simons, Barbara A. Spellman, Andrea H. Stoevenbelt, Barnabas Szaszi, Darinka Trübutschek, Francis Tuerlinckx, Eric L. Uhlmann, Wolf Vanpaemel, Jelte Wicherts, Eric Jan Wagenmakers

^*Corresponding author for this work

Research output: Contribution to Journal › Article › Academic › peer-review

Abstract

Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

Original language	English
Article number	240125
Pages (from-to)	1-28
Number of pages	28
Journal	Royal Society Open Science
Volume	11
Issue number	7
Early online date	24 Jul 2024
DOIs	https://doi.org/10.1098/rsos.240125
Publication status	Published - Jul 2024

Bibliographical note

Publisher Copyright:
© 2024 The Authors.

Funding

We thank Fr\u00E9d\u00E9rique Heppel for her support in creating the initial item list of the SEES and helping with data collection. This work was supported by an Amsterdam Brain and Cognition (ABC) project grant to A.S. (ABC PG 22). T.A. was supported in part by NSF IIS-1901386, NSF CAREER IIS-2142794. N.A.B. is supported by the German Research Foundation (DFG; BU 2400/11-1) and by the DFG priority programme \u2018META-REP: A Meta-scientific Programme to Analyse and Optimize Replicability in the Behavioural, Social, and Cognitive Sciences\u2019. B.D. was supported by the National Institute of General Medical Sciences of the National Institutes of Health under Award Number P20GM104420. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. N.N.N.v.D. was supported by the NWO Vici grant no. 181.029. A.D. was supported by the Knut and Alice Wallenberg Foundation, the Marianne and Marcus Wallenberg Foundation and the Jan Wallander and Tom Hedelius Foundation. D.M. is supported by a Vidi grant (VI.Vidi.191.091) from the Dutch Research Council (NWO). A.J.M. is supported by a Vici grant (016.Vici.185.068) from the Dutch Research Council (NWO). D.v.R. is supported by a Vidi grant (016.Vidi.188.001) from the Dutch Research Council (NWO). D.R.S. was supported by a grant from the UK Economic and Social Research Council. D.T. was supported by Marie Sklodowska-Curie grant agreement no. 101023805 under the European Union\u2019s Horizon 2020 research and innovation programme. E.L.U. was supported by an R&D grant from INSEAD. J.W. is supported by a VICI grant (VI.C.221.100) from the Dutch Research Council (NWO). N.N.N.v.D. was supported by the NWO Vici grant no. 181.029. A.D. was supported by the Knut and Alice Wallenberg Foundation, the Marianne and Marcus Wallenberg Foundation and the Jan Wallander and Tom Hedelius Foundation. D.M. is supported by a Vidi grant (VI.Vidi.191.091) from the Dutch Research Council (NWO). A.J.M. is supported by a Vici grant (016.Vici.185.068) from the Dutch Research Council (NWO). D.v.R. is supported by a Vidi grant (016.Vidi.188.001) from the Dutch Research Council (NWO). D.R.S. was supported by a grant from the UK Economic and Social Research Council. D.T. was supported by Marie Sklodowska-Curie grant agreement no. 101023805 under the European Union s Horizon 2020 research and innovation programme. E.L.U. was supported by an R&D grant from INSEAD. J.W. is supported by a VICI grant (VI.C.221.100) from the Dutch Research Council (NWO).

Funders	Funder number
Economic and Social Research Council
European Union s Horizon 2020 research and innovation programme
National Institute of General Medical Sciences
Institut Européen d'Administration des Affaires
Knut och Alice Wallenbergs Stiftelse
Horizon 2020 Framework Programme
Marcus och Amalia Wallenbergs minnesfond
National Institutes of Health	P20GM104420
Not added	VI.C.221.100, 181.029
Jan Wallanders och Tom Hedelius Stiftelse samt Tore Browaldhs Stiftelse	VI.Vidi.191.091
Deutsche Forschungsgemeinschaft	BU 2400/11-1
National Science Foundation	IIS-1901386, IIS-2142794
H2020 Marie Skłodowska-Curie Actions	101023805
Amsterdam Brain and Cognition	ABC PG 22

Keywords

crowdsourcing analysis
metascience
open science
scientific transparency
team science

Access to Document

10.1098/rsos.240125

Persistent URL (handle)

Persistent link

Cite this

Sarafoglou, A., Hoogeveen, S., Van Den Bergh, D., Aczel, B., Albers, C. J., Althoff, T., Botvinik-Nezer, R., Busch, N. A., Cataldo, A. M., Devezer, B., Van Dongen, N. N. N., Dreber, A., Fried, E. I., Hoekstra, R., Hoffman, S., Holzmeister, F., Huber, J., Huntington-Klein, N., Ioannidis, J., ... Wagenmakers, E. J. (2024). Subjective evidence evaluation survey for many-analysts studies. Royal Society Open Science, 11(7), 1-28. Article 240125. https://doi.org/10.1098/rsos.240125

@article{5c0196df0115464597878af24d309507,

title = "Subjective evidence evaluation survey for many-analysts studies",

abstract = "Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.",

keywords = "crowdsourcing analysis, metascience, open science, scientific transparency, team science",

author = "Alexandra Sarafoglou and Suzanne Hoogeveen and \{Van Den Bergh\}, Don and Balazs Aczel and Albers, \{Casper J.\} and Tim Althoff and Rotem Botvinik-Nezer and Busch, \{Niko A.\} and Cataldo, \{Andrea M.\} and Berna Devezer and \{Van Dongen\}, \{Noah N.N.\} and Anna Dreber and Fried, \{Eiko I.\} and Rink Hoekstra and Sabine Hoffman and Felix Holzmeister and J{\"u}rgen Huber and Nick Huntington-Klein and John Ioannidis and Magnus Johannesson and Michael Kirchler and Eric Loken and Mangin, \{Jan Francois\} and Dora Matzke and Menkveld, \{Albert J.\} and Gustav Nilsonne and \{Van Ravenzwaaij\}, Don and Martin Schweinsberg and Hannah Schulz-Kuempel and Shanks, \{David R.\} and Simons, \{Daniel J.\} and Spellman, \{Barbara A.\} and Stoevenbelt, \{Andrea H.\} and Barnabas Szaszi and Darinka Tr{\"u}butschek and Francis Tuerlinckx and Uhlmann, \{Eric L.\} and Wolf Vanpaemel and Jelte Wicherts and Wagenmakers, \{Eric Jan\}",

note = "Publisher Copyright: {\textcopyright} 2024 The Authors.",

year = "2024",

month = jul,

doi = "10.1098/rsos.240125",

language = "English",

volume = "11",

pages = "1--28",

journal = "Royal Society Open Science",

issn = "2054-5703",

publisher = "The Royal Society",

number = "7",

}

Sarafoglou, A, Hoogeveen, S, Van Den Bergh, D, Aczel, B, Albers, CJ, Althoff, T, Botvinik-Nezer, R, Busch, NA, Cataldo, AM, Devezer, B, Van Dongen, NNN, Dreber, A, Fried, EI, Hoekstra, R, Hoffman, S, Holzmeister, F, Huber, J, Huntington-Klein, N, Ioannidis, J, Johannesson, M, Kirchler, M, Loken, E, Mangin, JF, Matzke, D, Menkveld, AJ, Nilsonne, G, Van Ravenzwaaij, D, Schweinsberg, M, Schulz-Kuempel, H, Shanks, DR, Simons, DJ, Spellman, BA, Stoevenbelt, AH, Szaszi, B, Trübutschek, D, Tuerlinckx, F, Uhlmann, EL, Vanpaemel, W, Wicherts, J & Wagenmakers, EJ 2024, 'Subjective evidence evaluation survey for many-analysts studies', Royal Society Open Science, vol. 11, no. 7, 240125, pp. 1-28. https://doi.org/10.1098/rsos.240125

TY - JOUR

T1 - Subjective evidence evaluation survey for many-analysts studies

AU - Sarafoglou, Alexandra

AU - Hoogeveen, Suzanne

AU - Van Den Bergh, Don

AU - Aczel, Balazs

AU - Albers, Casper J.

AU - Althoff, Tim

AU - Botvinik-Nezer, Rotem

AU - Busch, Niko A.

AU - Cataldo, Andrea M.

AU - Devezer, Berna

AU - Van Dongen, Noah N.N.

AU - Dreber, Anna

AU - Fried, Eiko I.

AU - Hoekstra, Rink

AU - Hoffman, Sabine

AU - Holzmeister, Felix

AU - Huber, Jürgen

AU - Huntington-Klein, Nick

AU - Ioannidis, John

AU - Johannesson, Magnus

AU - Kirchler, Michael

AU - Loken, Eric

AU - Mangin, Jan Francois

AU - Matzke, Dora

AU - Menkveld, Albert J.

AU - Nilsonne, Gustav

AU - Van Ravenzwaaij, Don

AU - Schweinsberg, Martin

AU - Schulz-Kuempel, Hannah

AU - Shanks, David R.

AU - Simons, Daniel J.

AU - Spellman, Barbara A.

AU - Stoevenbelt, Andrea H.

AU - Szaszi, Barnabas

AU - Trübutschek, Darinka

AU - Tuerlinckx, Francis

AU - Uhlmann, Eric L.

AU - Vanpaemel, Wolf

AU - Wicherts, Jelte

AU - Wagenmakers, Eric Jan

PY - 2024/7

Y1 - 2024/7

N2 - Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

AB - Many-analysts studies explore how well an empirical claim withstands plausible alternative analyses of the same dataset by multiple, independent analysis teams. Conclusions from these studies typically rely on a single outcome metric (e.g. effect size) provided by each analysis team. Although informative about the range of plausible effects in a dataset, a single effect size from each team does not provide a complete, nuanced understanding of how analysis choices are related to the outcome. We used the Delphi consensus technique with input from 37 experts to develop an 18-item subjective evidence evaluation survey (SEES) to evaluate how each analysis team views the methodological appropriateness of the research design and the strength of evidence for the hypothesis. We illustrate the usefulness of the SEES in providing richer evidence assessment with pilot data from a previous many-analysts study.

KW - crowdsourcing analysis

KW - metascience

KW - open science

KW - scientific transparency

KW - team science

UR - https://www.scopus.com/pages/publications/85200718634

UR - https://www.scopus.com/inward/citedby.url?scp=85200718634&partnerID=8YFLogxK

U2 - 10.1098/rsos.240125

DO - 10.1098/rsos.240125

M3 - Article

AN - SCOPUS:85200718634

SN - 2054-5703

VL - 11

SP - 1

EP - 28

JO - Royal Society Open Science

JF - Royal Society Open Science

IS - 7

M1 - 240125

ER -

Subjective evidence evaluation survey for many-analysts studies

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Cite this