Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science

I.E. Vermeulen, C.J. Beukeboom, A.E. Batenburg, A.E. Avramiea, D.P. Stoyanov, R.N. van de Velde, D. Oegema

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p <.05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below.05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below.05. Results show 8.8% of p-values were misreported (74.5% too low). 1.3% of p-values were critically misreported, stating p <.05 while in fact p >.05 (88.3%) or vice versa (11.7%). Analyzing p-value frequencies just below.05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.
Original languageEnglish
Pages (from-to)253-279
JournalCommunication Methods and Measures
Volume9
Issue number4
Early online date30 Nov 2015
DOIs
Publication statusPublished - 2015

Fingerprint Dive into the research topics of 'Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science'. Together they form a unique fingerprint.

Cite this