How do we Evaluate Self-adaptive Software Systems? A Ten-Year Perspective of SEAMS

Ilias Gerostathopoulos, Thomas Vogel, Danny Weyns, Patricia Lago

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

With the increase of research in self-adaptive systems, there is a need to better understand the way research contributions are evaluated. Such insights will support researchers to better compare new findings when developing new knowledge for the community. However, so far there is no clear overview of how evaluations are performed in self-adaptive systems. To address this gap, we conduct a mapping study. The study focuses on experimental evaluations published in the last decade at the prime venue of research in software engineering for self-adaptive systems - the International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS). Results point out that specifics of self-adaptive systems require special attention in the experimental process, including the distinction of the managing system (i.e., the target of evaluation) and the managed system, the presence of uncertainties that affect the system behavior and hence need to be taken into account in data analysis, and the potential of managed systems to be reused across experiments, beyond replications. To conclude, we offer a set of suggestions derived from our study that can be used as input to enhance future experiments in self-adaptive systems.

Original languageEnglish
Title of host publication2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems (SEAMS)
Subtitle of host publicationProceedings
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages59-70
Number of pages12
ISBN (Electronic)9781665402897
DOIs
Publication statusPublished - 29 Jun 2021
Event2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2021 - Virtual, Online
Duration: 18 May 202124 May 2021

Conference

Conference2021 International Symposium on Software Engineering for Adaptive and Self-Managing Systems, SEAMS 2021
CityVirtual, Online
Period18/05/2124/05/21

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

Keywords

  • evaluation
  • mapping study
  • self adaptive systems

Fingerprint

Dive into the research topics of 'How do we Evaluate Self-adaptive Software Systems? A Ten-Year Perspective of SEAMS'. Together they form a unique fingerprint.

Cite this