Abstract
Background: In recent years, a discourse on how to systematically consider and report threats to validity started to gain momentum within the empirical software engineering community. Aims: With this study, we aim to systematically underpin the current state of threats to validity practices in software engineering research. Method: We conduct a literature review comprising 91 papers awarded with the ACM SIGSOFT Distinguished Paper Award at the ACM/IEEE International Conference on Software Engineering. Data is extracted and analyzed by considering six main facets of threats to validity, e.g., their explicit documentation, categorization, discussion of limitations, and trade-offs. Results: Results corroborate current critiques to the threats management state of the art. Threats result to be seldom discussed in depth, and are mostly considered as an enforced afterthought rather than an active concern of the research design and execution. Conclusions: To improve the observed practice, we derived items to consider for researchers, reviewers and readers, and call for a community action to increase the understanding of knowledge creation in empirical software engineering research.
Original language | English |
---|---|
Title of host publication | ESEM '24: Proceedings of the 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement |
Publisher | IEEE Computer Society |
Pages | 314-324 |
Number of pages | 11 |
ISBN (Electronic) | 9798400710476 |
DOIs | |
Publication status | Published - 2024 |
Event | 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024 - Barcelona, Spain Duration: 24 Oct 2024 → 25 Oct 2024 |
Publication series
Name | International Symposium on Empirical Software Engineering and Measurement |
---|---|
ISSN (Print) | 1949-3770 |
ISSN (Electronic) | 1949-3789 |
Conference
Conference | 18th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, ESEM 2024 |
---|---|
Country/Territory | Spain |
City | Barcelona |
Period | 24/10/24 → 25/10/24 |
Bibliographical note
Publisher Copyright:© 2024 Owner/Author.
Keywords
- Empirical studies
- Limitations
- Literature study
- Threats to validity