A probabilistic evaluation procedure for process model matching techniques

Elena Kuss*, Henrik Leopold, Han van der Aa, Heiner Stuckenschmidt, Hajo A. Reijers

*Corresponding author for this work

Research output: Contribution to JournalArticleAcademicpeer-review

Abstract

Process model matching refers to the automatic identification of corresponding activities between two process models. It represents the basis for many advanced process model analysis techniques such as the identification of similar process parts or process model search. A central problem is how to evaluate the performance of process model matching techniques. Current evaluation methods require a binary gold standard that clearly defines which correspondences are correct. The problem is that often not even humans can agree on a set of correct correspondences. Hence, evaluating the performance of matching techniques based on a binary gold standard does not take the true complexity of the matching problem into account and does not fairly assess the capabilities of a matching technique. In this paper, we propose a novel evaluation procedure for process model matching techniques. In particular, we build on the assessments of multiple annotators to define the notion of a non-binary gold standard. In this way, we avoid the problem of agreeing on a single set of correct correspondences. Based on this non-binary gold standard, we introduce probabilistic versions of precision, recall, and F-measure as well as a distance-based performance measure. We use a dataset from the Process Model Matching Contest 2015 and a total of 16 matching systems to assess and compare the insights that can be obtained by using our evaluation procedure. We find that our probabilistic evaluation procedure allows us to gain more detailed insights into the performance of matching systems than a traditional evaluation based on a binary gold standard.

Original languageEnglish
Pages (from-to)393-406
Number of pages14
JournalData and Knowledge Engineering
Volume117
Early online date18 Apr 2018
DOIs
Publication statusPublished - 1 Sep 2018

Keywords

  • Evaluation techniques
  • Probabilistic evaluation
  • Process model matching

Fingerprint

Dive into the research topics of 'A probabilistic evaluation procedure for process model matching techniques'. Together they form a unique fingerprint.

Cite this