Abstract
Background. Systematic Reviews (SRs) are means to collect and synthesize evidence from the identification, analysis, and interpretation of multiple sources, or {\em primary studies}. To this aim, they use a well-defined methodology that should mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.
Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.
Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, %and software quality, and evaluated with human annotators.
Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77).
Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.
Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.
Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, %and software quality, and evaluated with human annotators.
Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77).
Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.
Original language | English |
---|---|
Pages (from-to) | 311-340 |
Number of pages | 30 |
Journal | Data Science |
Volume | 2 |
Issue number | 1-2 |
DOIs | |
Publication status | Published - 25 Nov 2019 |
Keywords
- software architecture
- Empirical research
VU Research Profile
- Connected World
- Science for Sustainability