Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science

I.E. Vermeulen, C.J. Beukeboom, A.E. Batenburg, A.E. Avramiea, D.P. Stoyanov, R.N. van de Velde, D. Oegema

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p <.05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below.05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below.05. Results show 8.8% of p-values were misreported (74.5% too low). 1.3% of p-values were critically misreported, stating p <.05 while in fact p >.05 (88.3%) or vice versa (11.7%). Analyzing p-value frequencies just below.05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.
Original languageEnglish
Pages (from-to)253-279
JournalCommunication Methods and Measures
Volume9
Issue number4
Early online date30 Nov 2015
DOIs
Publication statusPublished - 2015

Fingerprint

communication sciences
statistical significance
cause
Communication
trend
Values
social psychology

Cite this

@article{b3e9469246b7438f9954abcfe0f1ddee,
title = "Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science",
abstract = "Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p <.05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below.05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below.05. Results show 8.8{\%} of p-values were misreported (74.5{\%} too low). 1.3{\%} of p-values were critically misreported, stating p <.05 while in fact p >.05 (88.3{\%}) or vice versa (11.7{\%}). Analyzing p-value frequencies just below.05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.",
author = "I.E. Vermeulen and C.J. Beukeboom and A.E. Batenburg and A.E. Avramiea and D.P. Stoyanov and {van de Velde}, R.N. and D. Oegema",
year = "2015",
doi = "10.1080/19312458.2015.1096333",
language = "English",
volume = "9",
pages = "253--279",
journal = "Communication Methods and Measures",
issn = "1931-2458",
publisher = "Routledge",
number = "4",

}

Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science. / Vermeulen, I.E.; Beukeboom, C.J.; Batenburg, A.E.; Avramiea, A.E.; Stoyanov, D.P.; van de Velde, R.N.; Oegema, D.

In: Communication Methods and Measures, Vol. 9, No. 4, 2015, p. 253-279.

Research output: Contribution to JournalArticleAcademicpeer-review

TY - JOUR

T1 - Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science

AU - Vermeulen, I.E.

AU - Beukeboom, C.J.

AU - Batenburg, A.E.

AU - Avramiea, A.E.

AU - Stoyanov, D.P.

AU - van de Velde, R.N.

AU - Oegema, D.

PY - 2015

Y1 - 2015

N2 - Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p <.05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below.05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below.05. Results show 8.8% of p-values were misreported (74.5% too low). 1.3% of p-values were critically misreported, stating p <.05 while in fact p >.05 (88.3%) or vice versa (11.7%). Analyzing p-value frequencies just below.05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.

AB - Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p <.05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below.05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below.05. Results show 8.8% of p-values were misreported (74.5% too low). 1.3% of p-values were critically misreported, stating p <.05 while in fact p >.05 (88.3%) or vice versa (11.7%). Analyzing p-value frequencies just below.05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.

U2 - 10.1080/19312458.2015.1096333

DO - 10.1080/19312458.2015.1096333

M3 - Article

VL - 9

SP - 253

EP - 279

JO - Communication Methods and Measures

JF - Communication Methods and Measures

SN - 1931-2458

IS - 4

ER -