Extreme and Implausible Effect Sizes in Meta-Analyses of Psychological Treatments: Meta-Epidemiological Study and Development of a Simple Flagging Tool (Materials & Code)

In meta-analyses of psychological treatments, all eligible studies are typically included by default, even those reporting extreme and barely plausible effects. Random-effects models are the standard in mental health research, but assign disproportionate weight to such outliers, and may strongly overestimate effect heterogeneity. Existing guidance allows otherwise eligible trials to be excluded from meta-analyses altogether, but only if there are clear and justifiable criteria. As of now, there is no consensus about when effect sizes of psychological treatment can be considered too extreme or implausible to merit inclusion in a meta-analysis. In this study, we developed a simple, empirically derived flagging system to identify potentially implausible effect sizes, based on (i) the magnitude of the effect, (ii) the study’s actual power, and (iii) its overall quality or risk of bias. Thresholds were established using a large database of clinical trials on psychological treatments across 12 indications (>2,000 effect sizes). In practice, this heuristic may support the exclusion of implausible results from meta-analyses, prompt further plausibility checks, or aid in assessing the credibility of new trial findings. Finally, we investigated the characteristics of flagged studies in the Metapsy databases and assessed how their exclusion affects pooled estimates of psychological treatment effects.