Reducing the Effort for Systematic Reviews in Software Engineering

Francesco Osborne, Henry Muccini, P. Lago, Enrico Motta

Research output: Online publication or Non-textual formOnline publication or WebsiteAcademic

Abstract

Background. Systematic Reviews (SRs) are means to collect and synthesize evidence from the identification, analysis, and interpretation of multiple sources, or {\em primary studies}. To this aim, they use a well-defined methodology that should mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.

Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.

Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, %and software quality, and evaluated with human annotators.

Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77).

Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.
Original languageEnglish
Publication statusPublished - 2018

Fingerprint

Software engineering
Software architecture
Ontology
Automation
Scalability
Semantics

Keywords

  • software architecture
  • Empirical research

VU Research Profile

  • Connected World
  • Science for Sustainability

Cite this

Osborne, F. (Author), Muccini, H. (Author), Lago, P. (Author), & Motta, E. (Author). (2018). Reducing the Effort for Systematic Reviews in Software Engineering. Online publication or Website
Osborne, Francesco (Author) ; Muccini, Henry (Author) ; Lago, P. (Author) ; Motta, Enrico (Author). / Reducing the Effort for Systematic Reviews in Software Engineering. [Online publication or Website].
@misc{86b177d3ca584d0ab7dbc932e74445c0,
title = "Reducing the Effort for Systematic Reviews in Software Engineering",
abstract = "Background. Systematic Reviews (SRs) are means to collect and synthesize evidence from the identification, analysis, and interpretation of multiple sources, or {\em primary studies}. To this aim, they use a well-defined methodology that should mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, {\%}and software quality, and evaluated with human annotators.Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77). Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.",
keywords = "software architecture, Empirical research",
author = "Francesco Osborne and Henry Muccini and P. Lago and Enrico Motta",
year = "2018",
language = "English",

}

Osborne, F, Muccini, H, Lago, P & Motta, E, Reducing the Effort for Systematic Reviews in Software Engineering, 2018, Online publication or Website.
Reducing the Effort for Systematic Reviews in Software Engineering. Osborne, Francesco (Author); Muccini, Henry (Author); Lago, P. (Author); Motta, Enrico (Author). 2018.

Research output: Online publication or Non-textual formOnline publication or WebsiteAcademic

TY - ADVS

T1 - Reducing the Effort for Systematic Reviews in Software Engineering

AU - Osborne, Francesco

AU - Muccini, Henry

AU - Lago, P.

AU - Motta, Enrico

PY - 2018

Y1 - 2018

N2 - Background. Systematic Reviews (SRs) are means to collect and synthesize evidence from the identification, analysis, and interpretation of multiple sources, or {\em primary studies}. To this aim, they use a well-defined methodology that should mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, %and software quality, and evaluated with human annotators.Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77). Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.

AB - Background. Systematic Reviews (SRs) are means to collect and synthesize evidence from the identification, analysis, and interpretation of multiple sources, or {\em primary studies}. To this aim, they use a well-defined methodology that should mitigate the risks of biases and ensure repeatability for later updates. SRs, however, involve significant effort.Goal. The goal of this paper is to introduce a novel expert-driven automatic methodology (EDAM) that, among other benefits, while taking advantage of the value provided by human expertise, reduces the number of manual tedious tasks involved in SRs.Method. Starting from current methodologies for SRs, we replaced the steps of keywording and data extraction with an automatic methodology for generating a domain ontology and classifying the primary studies. This methodology has been then applied in the software engineering sub-area of software architecture, %and software quality, and evaluated with human annotators.Results. The result is a novel expert-driven automatic methodology for performing SRs. This combines ontology-learning techniques and semantic technologies with the human-in-the-loop. The first (thanks to automation) fosters scalability, objectivity, reproducibility and granularity of the studies; the second allows tailoring to the specific focus of the study at hand, as well as knowledge reuse from domain experts. We evaluated EDAM on the field of Software Architecture and found that its performance in classifying papers were not statistically significant different from the ones of six senior researchers (p=0.77). Conclusions. Thanks to automation of the less creative steps in SRs, our methodology allows researchers to skip the tedious tasks of keywording and manually classifying primary studies, thus freeing effort for the analysis and the discussion.

KW - software architecture

KW - Empirical research

M3 - Online publication or Website

ER -

Osborne F (Author), Muccini H (Author), Lago P (Author), Motta E (Author). Reducing the Effort for Systematic Reviews in Software Engineering 2018.