Systematic review search strategies are poorly reported and not reproducible: a cross-sectional meta-research study

Melissa L Rethlefsen, Tara J Brigham, Carrie Price, David Moher, Lex M Bouter, Jamie J Kirkham, Sara Schroter, Maurice P Zeegers

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

OBJECTIVE: To determine the reproducibility of biomedical systematic review search strategies.

STUDY DESIGN AND SETTING: A cross-sectional reproducibility study was conducted on a random sample of 100 systematic reviews indexed in MEDLINE in November 2021. The primary outcome measure is the percentage of systematic reviews for which all database searches can be reproduced, operationalized as fulfilling 6 key PRISMA-S reporting guideline items and having all database searches reproduced within 10% of the number of original results. Key reporting guideline items included database name, multi-database searching, full search strategies, limits and restrictions, date(s) of searches, and total records.

RESULTS: The 100 systematic review articles contained 453 database searches. Only 22 (4.9%) database searches reported all six PRISMA-S items. Forty-seven (10.4%) database searches could be reproduced within 10% of the number of results from the original search; 6 searches differed by more than 1000% between the originally reported number of results and the reproduction. Only one systematic review article provided the necessary search details to be fully reproducible.

CONCLUSION: Systematic review search reporting is poor. To correct this will require a multi-faceted response from authors, peer reviewers, journal editors, and database providers.

Original languageEnglish
Article number111229
Pages (from-to)1-12
Number of pages12
JournalJournal of clinical epidemiology
Volume166
Early online date3 Dec 2023
DOIs
Publication statusPublished - Feb 2024

Bibliographical note

Copyright © 2023 The Author(s). Published by Elsevier Inc. All rights reserved.

Funding

No specific funding was received for this study. It is part of M.L.R.’s self-funded PhD project registered at Maastricht University, the Netherlands, in collaboration with the BMJ, United Kingdom. M.L.R. and D.M. are co-authors of PRISMA-S, which is used in this study. T.J.B., C.P., L.M.B., and M.P.Z. have no competing interests. S.S. is a full-time employee of BMJ. J.J.K. is a statistical reviewer for BMJ. D.M. is an editorial board member of Journal of Clinical Epidemiology.

FundersFunder number
BMJ

    Fingerprint

    Dive into the research topics of 'Systematic review search strategies are poorly reported and not reproducible: a cross-sectional meta-research study'. Together they form a unique fingerprint.

    Cite this