Implausible Effects of Psychological Interventions Meta-Epidemiological Study and Development of a Simple Flagging Tool

In meta-analyses of psychological interventions, trials occasionally report effects that appear implausibly large. While such results are unlikely to reflect genuine treatment effects, this is rarely verifiable, and studies are often retained in the meta-analytic evidence. Existing guidance allows for highly questionable results to be discarded in evidence syntheses, but offers little direction on how to identify them in practice. Consequently, it remains unclear to what extent suspicious evidence has biased effect estimates in psychological intervention research. In this study, we develop a simple flagging tool to detect such trials, based on the (1) compatibility of their effect size with low risk of bias evidence, (2) achieved power, and (3) methodological rigor. We also examine specific characteristics of studies flagged by this tool, and the impact of their exclusion on pooled estimates and heterogeneity. In total, 2,881 effect sizes from 1,246 randomized trials were included from twelve living databases of psychological interventions for mental health problems. Overall, 5.3n=153 across 102 studies) were flagged. Reanalysis of 135 meta-analyses from a large-scale evaluation of psychological interventions showed that excluding flagged studies led to substantially lower effect estimates (reductions of up to 31.2 and decreased between-study heterogeneity (up to 51.1 indication-wide analyses). The flagging tool has been integrated into the open-source R package textquotedblleftmetapsyToolstextquotedblright. We discuss potential explanations for the accumulation of improbable findings in the published literature, and how the application of our tool may strengthen quality control in meta-analytic research.Question How can implausibly large effect sizes in psychological intervention trials be identified, and what impact do they have on meta-analytic evidence?Findings In this meta-epidemiological study, we developed a simple flagging tool based on effect size compatibility, statistical power, and methodological rigor. Applying it to 2,881 effect sizes from 12 living databases, we found that 5.3reductions of up to 31.2 and decreased between-study heterogeneity (up to 51.1.Meaning Implausible effects can substantially bias meta-analytic evidence. A simple flagging tool can help identify such effects and improve quality control in evidence syntheses. The tool has been integrated into an open-source R package for routine use.Competing Interest StatementDP was funded by the Horizon-MSCA-2021-PF-01 research program of the European Union under Grant Agreement No.101061648. The funder had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the article; and decision to submit the article for publication. TAF reports per-sonal fees from Boehringer-Ingelheim, Daiichi Sankyo, DT Axis, Micron, Shionogi, SONY, and UpToDate, and a grant from DT Axis and Shionogi, outside the submitted work. All other authors report no potential conflicts of interest.Funding StatementNo external funding.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:The study used oly openly available data in published trial reports, as collected by the Metapsy living meta-analytic databases.I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesData and analysis code are available at: 10.5281/zenodo.17143121