Reducing the Effort for Systematic Reviews in Software Engineering

Francesco Osborne, Henry Muccini, P. Lago, Enrico Motta

Research output: Contribution to JournalArticleAcademicpeer-review

207 Downloads (Pure)


Background. Systematic Reviews (SRs) are means to collect and synthesize evidence from the identification, analysis, and interpretation of multiple sources, or {\em primary studies}. To this aim, they use a well-defined methodology that should mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.

Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.

Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, %and software quality, and evaluated with human annotators.

Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77).

Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.
Original languageEnglish
Pages (from-to)311-340
Number of pages30
JournalData Science
Issue number1-2
Publication statusPublished - 25 Nov 2019


  • software architecture
  • Empirical research

VU Research Profile

  • Connected World
  • Science for Sustainability


Dive into the research topics of 'Reducing the Effort for Systematic Reviews in Software Engineering'. Together they form a unique fingerprint.

Cite this