TY - GEN
T1 - Seven commandments for benchmarking semantic flow processing systems
AU - Scharrenbach, Thomas
AU - Urbani, Jacopo
AU - Margara, Alessandro
AU - Della Valle, Emanuele
AU - Bernstein, Abraham
PY - 2013/10/9
Y1 - 2013/10/9
N2 - Over the last few years, the processing of dynamic data has gained increasing attention in the Semantic Web community. This led to the development of several stream reasoning systems that enable on-the-fly processing of semantically annotated data that changes over time. Due to their streaming nature, analyzing such systems is extremely difficult. Currently, their evaluation is conducted under heterogeneous scenarios, hampering their comparison and an understanding of their benefits and limitations. In this paper, we strive for a better understanding of the key challenges that these systems must face and define a generic methodology to evaluate their performance. Specifically, we identify three Key Performance Indicators and seven commandments that specify how to design the stress tests for system evaluation.
AB - Over the last few years, the processing of dynamic data has gained increasing attention in the Semantic Web community. This led to the development of several stream reasoning systems that enable on-the-fly processing of semantically annotated data that changes over time. Due to their streaming nature, analyzing such systems is extremely difficult. Currently, their evaluation is conducted under heterogeneous scenarios, hampering their comparison and an understanding of their benefits and limitations. In this paper, we strive for a better understanding of the key challenges that these systems must face and define a generic methodology to evaluate their performance. Specifically, we identify three Key Performance Indicators and seven commandments that specify how to design the stress tests for system evaluation.
UR - http://www.scopus.com/inward/record.url?scp=84885017185&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=84885017185&partnerID=8YFLogxK
U2 - https://link.springer.com/chapter/10.1007/978-3-642-38288-8_21
DO - https://link.springer.com/chapter/10.1007/978-3-642-38288-8_21
M3 - Conference contribution
AN - SCOPUS:84885017185
SN - 9783642382871
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 305
EP - 319
BT - The Semantic Web
T2 - 10th International Conference on The Semantic Web: Semantics and Big Data, ESWC 2013
Y2 - 26 May 2013 through 30 May 2013
ER -