Reproducibility and replicability of rodent phenotyping in preclinical studies

Neri Kafkafi; Joseph Agassi; Elissa J. Chesler; John C. Crabbe; Wim E. Crusio; David Eilam; Robert Gerlai; Ilan Golani; Alex Gomez-Marin; Ruth Heller; Fuad Iraqi; Iman Jaljuli; Natasha A. Karp; Hugh Morgan; George Nicholson; Donald W. Pfaff; S. Helene Richter; Philip B. Stark; Oliver Stiedl; Victoria Stodden; Lisa M. Tarantino; Valter Tucci; William Valdar; Robert W. Williams; Hanno Würbel; Yoav Benjamini

doi:10.1016/j.neubiorev.2018.01.003

Reproducibility and replicability of rodent phenotyping in preclinical studies

Neri Kafkafi^*, Joseph Agassi, Elissa J. Chesler, John C. Crabbe, Wim E. Crusio, David Eilam, Robert Gerlai, Ilan Golani, Alex Gomez-Marin, Ruth Heller, Fuad Iraqi, Iman Jaljuli, Natasha A. Karp, Hugh Morgan, George Nicholson, Donald W. Pfaff, S. Helene Richter, Philip B. Stark, Oliver Stiedl, Victoria StoddenLisa M. Tarantino, Valter Tucci, William Valdar, Robert W. Williams, Hanno Würbel, Yoav Benjamini

^*Corresponding author for this work

Research output: Contribution to Journal › Review article › Academic › peer-review

Abstract

The scientific community is increasingly concerned with the proportion of published “discoveries” that are not replicated in subsequent studies. The field of rodent behavioral phenotyping was one of the first to raise this concern, and to relate it to other methodological issues: the complex interaction between genotype and environment; the definitions of behavioral constructs; and the use of laboratory mice and rats as model species for investigating human health and disease mechanisms. In January 2015, researchers from various disciplines gathered at Tel Aviv University to discuss these issues. The general consensus was that the issue is prevalent and of concern, and should be addressed at the statistical, methodological and policy levels, but is not so severe as to call into question the validity and the usefulness of model organisms as a whole. Well-organized community efforts, coupled with improved data and metadata sharing, have a key role in identifying specific problems and promoting effective solutions. Replicability is closely related to validity, may affect generalizability and translation of findings, and has important ethical implications.

Original language	English
Pages (from-to)	218-232
Number of pages	15
Journal	Neuroscience and Biobehavioral Reviews
Volume	87
Issue number	April
Early online date	31 Jan 2018
DOIs	https://doi.org/10.1016/j.neubiorev.2018.01.003
Publication status	Published - Apr 2018

Funding

This work is supported by European Research Council grants FP7/2007-2013 ERC agreement no [294519]-PSARPS (YB, NK, IG, IJ), by NIH AA018776 (EJC), by NIH DA045401 (YB, NK) and by the Spanish Ministry of Economy and Competitivity (Severo Ochoa Center of Excellence programme SEV2013-0317 start-up funds to AGM). We thank Prof. Allan Kalueff for many useful comments. This paper is the result of a scientific meeting in Tel Aviv University. Videos of the full lectures can be accessed at the following links. 1. Yoav Benjamini: “Assessing Replicability in One Own's Lab: A Community Effort” https://www.youtube.com/watch?v=y-Q4GWkJicE&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=15 . 2. Elissa J. Chesler: “Finding Consilience in Genetic and Genomic Analyses of Behavior” https://www.youtube.com/watch?v=f9TdNXipRPQ&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=20 . 3. John C. Crabbe: “Managing Sources of E to Address GXE” https://www.youtube.com/watch?v=A7R2iZfjydA&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=4 . 4. Wim E. Crusio: “What do We Want to Reproduce?” https://www.youtube.com/watch?v=3ZuEZw8mDjY&index=6&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 5. David Eilam: “Variability: A By-Product Noise or an Essential Component of Motor Behavior” https://www.youtube.com/watch?v=ADgVHDFZpUg&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=7 . 6. Robert Gerlai: “Behavioral Phenotyping: The Double Edged Sword” https://www.youtube.com/watch?v=pRrNz0PTKc4&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=9 . 7. Ilan Golani: “Replicability as a Virtue” https://www.youtube.com/watch?v=KiRSxpA8qZY&index=10&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&spfreload=10 . 8. Alex Gomez-Marin: “Toward a Behavioral Homology Between Insects and Rodents” https://www.youtube.com/watch?v=Hu8PwdKap3k&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=25&spfreload=10 . 9. Ruth Heller: “Assessing Replicability Across Laboratories: The R-Value” https://www.youtube.com/watch?v=b2uGOp9yLMw&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=22 . 10. Fuad Iraqi: “TAU Collaborative Cross Mice a Powerful GRP for Dissection Host Susceptibility to Diseases” https://www.youtube.com/watch?v=NV62o7Ubrfg&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=21 . 11. Natasha A. Karp: “Simulation Studies to Investigate Workflow and its Impact on Reliability of Phenodeviant Calls: https://www.youtube.com/watch?v=N5wTrgXvsAY&index=17&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 12. Neri Kafkafi: “The Random Lab Model for Assessing the Replicability of Phenotyping Results Across Laboratories” https://www.youtube.com/watch?v=XsQawSBA6Vc&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=12 . 13. George Nicholson & Hugh Morgan: “The Empirical Reproducibility of High-Throughput Mouse Phenotyping” https://www.youtube.com/watch?v=KzrrP6_F8r8&index=18&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 14. Donald W. Pfaff: “Application of Strict Methodology and Applied Mathematical Statistics to Mouse Behavioral Data” https://www.youtube.com/watch?v=hB0FnO9evbY&index=8&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 15. Philip B. Stark: “Preproducibility for Research, Teaching, Collaboration, and Publishing” https://www.youtube.com/watch?v=wHryMtEBkB4&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=13 . 16. Oliver Stiedl: “Individuality of Avoidance Behavior of Mice in an Automated Home Cage Environment” https://www.youtube.com/watch?v=gUIgRW5luZY&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=24 . 17. Victoria Stodden: “Computational and Statistical Reproducibility” https://www.youtube.com/watch?v=GzrOcqz8TVY&index=14&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 18. Valter Tucci: “Phenotyping Behaviour Across Laboratories and Across Mouse Strains” https://www.youtube.com/watch?v=iTlsFaj62oQ&index=22&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 19. William Valdar & Lisa M. Tarantino: “The Effect of Genetic Background on Behavioral Variability: Implications for Replicability?” https://www.youtube.com/watch?v=63sgLO4Hd04&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP&index=5 . 20. Robert W. Williams: “Data Rescue for Replication: Finding, Annotating, and Reusing Data for the BXD Mouse Cohort” https://www.youtube.com/watch?v=goocssSA33g&index=19&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP . 21. Hanno Würbel & S. Helene Richter: “On Standardization and Other Fallacies in Animal Phenotyping” https://www.youtube.com/watch?v=tfW35740q3k&index=11&list=PLNiWLB_wsOg74GlfLNyAcTo-TshyAcdLP .

Funders	Funder number
National Institutes of Health	DA045401, AA018776
National Institute on Alcohol Abuse and Alcoholism	U01AA013519
Seventh Framework Programme	294519
European Research Council
Ministerio de Economía y Competitividad

Keywords

Data sharing
False discoveries
GxE interaction
Heterogenization
Replicability
Reproducibility
Validity

Access to Document

10.1016/j.neubiorev.2018.01.003

Persistent URL (handle)

Persistent link

Cite this

Kafkafi, N., Agassi, J., Chesler, E. J., Crabbe, J. C., Crusio, W. E., Eilam, D., Gerlai, R., Golani, I., Gomez-Marin, A., Heller, R., Iraqi, F., Jaljuli, I., Karp, N. A., Morgan, H., Nicholson, G., Pfaff, D. W., Richter, S. H., Stark, P. B., Stiedl, O., ... Benjamini, Y. (2018). Reproducibility and replicability of rodent phenotyping in preclinical studies. Neuroscience and Biobehavioral Reviews, 87(April), 218-232. https://doi.org/10.1016/j.neubiorev.2018.01.003

@article{0ba0311fe29148ec85b842bf57877ff2,

title = "Reproducibility and replicability of rodent phenotyping in preclinical studies",

abstract = "The scientific community is increasingly concerned with the proportion of published “discoveries” that are not replicated in subsequent studies. The field of rodent behavioral phenotyping was one of the first to raise this concern, and to relate it to other methodological issues: the complex interaction between genotype and environment; the definitions of behavioral constructs; and the use of laboratory mice and rats as model species for investigating human health and disease mechanisms. In January 2015, researchers from various disciplines gathered at Tel Aviv University to discuss these issues. The general consensus was that the issue is prevalent and of concern, and should be addressed at the statistical, methodological and policy levels, but is not so severe as to call into question the validity and the usefulness of model organisms as a whole. Well-organized community efforts, coupled with improved data and metadata sharing, have a key role in identifying specific problems and promoting effective solutions. Replicability is closely related to validity, may affect generalizability and translation of findings, and has important ethical implications.",

keywords = "Data sharing, False discoveries, GxE interaction, Heterogenization, Replicability, Reproducibility, Validity",

author = "Neri Kafkafi and Joseph Agassi and Chesler, {Elissa J.} and Crabbe, {John C.} and Crusio, {Wim E.} and David Eilam and Robert Gerlai and Ilan Golani and Alex Gomez-Marin and Ruth Heller and Fuad Iraqi and Iman Jaljuli and Karp, {Natasha A.} and Hugh Morgan and George Nicholson and Pfaff, {Donald W.} and Richter, {S. Helene} and Stark, {Philip B.} and Oliver Stiedl and Victoria Stodden and Tarantino, {Lisa M.} and Valter Tucci and William Valdar and Williams, {Robert W.} and Hanno W{\"u}rbel and Yoav Benjamini",

year = "2018",

month = apr,

doi = "10.1016/j.neubiorev.2018.01.003",

language = "English",

volume = "87",

pages = "218--232",

journal = "Neuroscience and Biobehavioral Reviews",

issn = "0149-7634",

publisher = "Elsevier Ltd",

number = "April",

}

Kafkafi, N, Agassi, J, Chesler, EJ, Crabbe, JC, Crusio, WE, Eilam, D, Gerlai, R, Golani, I, Gomez-Marin, A, Heller, R, Iraqi, F, Jaljuli, I, Karp, NA, Morgan, H, Nicholson, G, Pfaff, DW, Richter, SH, Stark, PB, Stiedl, O, Stodden, V, Tarantino, LM, Tucci, V, Valdar, W, Williams, RW, Würbel, H & Benjamini, Y 2018, 'Reproducibility and replicability of rodent phenotyping in preclinical studies', Neuroscience and Biobehavioral Reviews, vol. 87, no. April, pp. 218-232. https://doi.org/10.1016/j.neubiorev.2018.01.003

TY - JOUR

T1 - Reproducibility and replicability of rodent phenotyping in preclinical studies

AU - Kafkafi, Neri

AU - Agassi, Joseph

AU - Chesler, Elissa J.

AU - Crabbe, John C.

AU - Crusio, Wim E.

AU - Eilam, David

AU - Gerlai, Robert

AU - Golani, Ilan

AU - Gomez-Marin, Alex

AU - Heller, Ruth

AU - Iraqi, Fuad

AU - Jaljuli, Iman

AU - Karp, Natasha A.

AU - Morgan, Hugh

AU - Nicholson, George

AU - Pfaff, Donald W.

AU - Richter, S. Helene

AU - Stark, Philip B.

AU - Stiedl, Oliver

AU - Stodden, Victoria

AU - Tarantino, Lisa M.

AU - Tucci, Valter

AU - Valdar, William

AU - Williams, Robert W.

AU - Würbel, Hanno

AU - Benjamini, Yoav

PY - 2018/4

Y1 - 2018/4

N2 - The scientific community is increasingly concerned with the proportion of published “discoveries” that are not replicated in subsequent studies. The field of rodent behavioral phenotyping was one of the first to raise this concern, and to relate it to other methodological issues: the complex interaction between genotype and environment; the definitions of behavioral constructs; and the use of laboratory mice and rats as model species for investigating human health and disease mechanisms. In January 2015, researchers from various disciplines gathered at Tel Aviv University to discuss these issues. The general consensus was that the issue is prevalent and of concern, and should be addressed at the statistical, methodological and policy levels, but is not so severe as to call into question the validity and the usefulness of model organisms as a whole. Well-organized community efforts, coupled with improved data and metadata sharing, have a key role in identifying specific problems and promoting effective solutions. Replicability is closely related to validity, may affect generalizability and translation of findings, and has important ethical implications.

AB - The scientific community is increasingly concerned with the proportion of published “discoveries” that are not replicated in subsequent studies. The field of rodent behavioral phenotyping was one of the first to raise this concern, and to relate it to other methodological issues: the complex interaction between genotype and environment; the definitions of behavioral constructs; and the use of laboratory mice and rats as model species for investigating human health and disease mechanisms. In January 2015, researchers from various disciplines gathered at Tel Aviv University to discuss these issues. The general consensus was that the issue is prevalent and of concern, and should be addressed at the statistical, methodological and policy levels, but is not so severe as to call into question the validity and the usefulness of model organisms as a whole. Well-organized community efforts, coupled with improved data and metadata sharing, have a key role in identifying specific problems and promoting effective solutions. Replicability is closely related to validity, may affect generalizability and translation of findings, and has important ethical implications.

KW - Data sharing

KW - False discoveries

KW - GxE interaction

KW - Heterogenization

KW - Replicability

KW - Reproducibility

KW - Validity

UR - http://www.scopus.com/inward/record.url?scp=85042385750&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85042385750&partnerID=8YFLogxK

U2 - 10.1016/j.neubiorev.2018.01.003

DO - 10.1016/j.neubiorev.2018.01.003

M3 - Review article

AN - SCOPUS:85042385750

SN - 0149-7634

VL - 87

SP - 218

EP - 232

JO - Neuroscience and Biobehavioral Reviews

JF - Neuroscience and Biobehavioral Reviews

IS - April

ER -

Reproducibility and replicability of rodent phenotyping in preclinical studies

Abstract

Funding

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this