Threats to Validity in Software Engineering - hypocritical paper section or essential analysis?

Patricia Lago, Per Runeson, Qunying Song, Roberto Verdecchia

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Background: In recent years, a discourse on how to systematically consider and report threats to validity started to gain momentum within the empirical software engineering community. Aims: With this study, we aim to systematically underpin the current state of threats to validity practices in software engineering research. Method: We conduct a literature review comprising 91 papers awarded with the ACM SIGSOFT Distinguished Paper Award at the ACM/IEEE International Conference on Software Engineering. Data is extracted and analyzed by considering six main facets of threats to validity, e.g., their explicit documentation, categorization, discussion of limitations, and trade-offs. Results: Results corroborate current critiques to the threats management state of the art. Threats result to be seldom discussed in depth, and are mostly considered as an enforced afterthought rather than an active concern of the research design and execution. Conclusions: To improve the observed practice, we derived items to consider for researchers, reviewers and readers, and call for a community action to increase the understanding of knowledge creation in empirical software engineering research.

Original languageEnglish
Title of host publicationESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement
PublisherIEEE Computer Society
Pages314-324
Number of pages11
ISBN (Electronic)9798400710476
DOIs
Publication statusPublished - 2024
Event18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024 - Barcelona, Spain
Duration: 24 Oct 202425 Oct 2024

Publication series

NameInternational Symposium on Empirical Software Engineering and Measurement
ISSN (Print)1949-3770
ISSN (Electronic)1949-3789

Conference

Conference18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024
Country/TerritorySpain
CityBarcelona
Period24/10/2425/10/24

Bibliographical note

Publisher Copyright:
© 2024 Owner/Author.

Keywords

  • Empirical studies
  • Limitations
  • Literature study
  • Threats to validity

Fingerprint

Dive into the research topics of 'Threats to Validity in Software Engineering - hypocritical paper section or essential analysis?'. Together they form a unique fingerprint.

Cite this