Blinded by the light: How a focus on statistical “significance” may cause p-value misreporting and an excess of p-values just below .05 in communication science

Publication bias promotes papers providing “significant” findings, thus incentivizing researchers to produce such findings. Prior studies suggested that researchers’ focus on “p <.05” yields—intentional or unintentional—p-value misreporting, and excess p-values just below.05. To assess whether similar distortions occur in communication science, we extracted 5,834 test statistics from 693 recent communication science ISI papers, and assessed prevalence of p-values (1) misreported, and (2) just below.05. Results show 8.8% of p-values were misreported (74.5% too low). 1.3% of p-values were critically misreported, stating p <.05 while in fact p >.05 (88.3%) or vice versa (11.7%). Analyzing p-value frequencies just below.05 using a novel method did not unequivocally demonstrate “p-hacking”—excess p-values could be alternatively explained by (severe) publication bias. Results for 19,830 p-values from social psychology were strikingly similar. We conclude that publication bias, publication pressure, and verification bias distort the communication science knowledge base, and suggest solutions to this problem.