Lessons Learned from Five Years of Artifact Evaluations at EuroSys

Daniele Cono D’Elia; Thaleia Dimitra Doudali; Cristiano Giuffrida; Miguel Matos; Mathias Payer; Solal Pirelli; Georgios Portokalidis; Valerio Schiavoni; Salvatore Signorello; Anjo Vahldiek-Oberwagner

doi:10.1145/3736731.3746152

Lessons Learned from Five Years of Artifact Evaluations at EuroSys

Daniele Cono D’Elia^*, Thaleia Dimitra Doudali, Cristiano Giuffrida, Miguel Matos, Mathias Payer, Solal Pirelli, Georgios Portokalidis, Valerio Schiavoni, Salvatore Signorello, Anjo Vahldiek-Oberwagner

^*Corresponding author for this work

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.

Original language	English
Title of host publication	ACM REP '25: Proceedings of the 3rd ACM Conference on Reproducibility and Replicability
Publisher	Association for Computing Machinery, Inc
Pages	108-120
Number of pages	13
ISBN (Electronic)	9798400719585
DOIs	https://doi.org/10.1145/3736731.3746152
Publication status	Published - 2025
Event	3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025 - Vancouver, Canada Duration: 29 Jul 2025 → 31 Jul 2025

Conference

Conference	3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025
Country/Territory	Canada
City	Vancouver
Period	29/07/25 → 31/07/25

Bibliographical note

Publisher Copyright:
© 2025 Copyright held by the owner/author(s).

Keywords

Artifact Evaluation
Conference-scale artifact evaluation experiences
Conference-scale artifact evaluation practices
Reproducibility

Access to Document

10.1145/3736731.3746152Licence: CC BY

Persistent URL (handle)

Persistent link

Cite this

D’Elia, D. C., Doudali, T. D., Giuffrida, C., Matos, M., Payer, M., Pirelli, S., Portokalidis, G., Schiavoni, V., Signorello, S., & Vahldiek-Oberwagner, A. (2025). Lessons Learned from Five Years of Artifact Evaluations at EuroSys. In ACM REP '25: Proceedings of the 3rd ACM Conference on Reproducibility and Replicability (pp. 108-120). Association for Computing Machinery, Inc. https://doi.org/10.1145/3736731.3746152

@inproceedings{3f52b121a5454e3294b09290b160786f,

title = "Lessons Learned from Five Years of Artifact Evaluations at EuroSys",

abstract = "Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.",

keywords = "Artifact Evaluation, Conference-scale artifact evaluation experiences, Conference-scale artifact evaluation practices, Reproducibility",

author = "D{\textquoteright}Elia, \{Daniele Cono\} and Doudali, \{Thaleia Dimitra\} and Cristiano Giuffrida and Miguel Matos and Mathias Payer and Solal Pirelli and Georgios Portokalidis and Valerio Schiavoni and Salvatore Signorello and Anjo Vahldiek-Oberwagner",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025 ; Conference date: 29-07-2025 Through 31-07-2025",

year = "2025",

doi = "10.1145/3736731.3746152",

language = "English",

pages = "108--120",

booktitle = "ACM REP '25: Proceedings of the 3rd ACM Conference on Reproducibility and Replicability",

publisher = "Association for Computing Machinery, Inc",

}

D’Elia, DC, Doudali, TD, Giuffrida, C, Matos, M, Payer, M, Pirelli, S, Portokalidis, G, Schiavoni, V, Signorello, S & Vahldiek-Oberwagner, A 2025, Lessons Learned from Five Years of Artifact Evaluations at EuroSys. in ACM REP '25: Proceedings of the 3rd ACM Conference on Reproducibility and Replicability. Association for Computing Machinery, Inc, pp. 108-120, 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025, Vancouver, Canada, 29/07/25. https://doi.org/10.1145/3736731.3746152

Lessons Learned from Five Years of Artifact Evaluations at EuroSys. / D’Elia, Daniele Cono; Doudali, Thaleia Dimitra; Giuffrida, Cristiano et al.
ACM REP '25: Proceedings of the 3rd ACM Conference on Reproducibility and Replicability. Association for Computing Machinery, Inc, 2025. p. 108-120.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Lessons Learned from Five Years of Artifact Evaluations at EuroSys

AU - D’Elia, Daniele Cono

AU - Doudali, Thaleia Dimitra

AU - Giuffrida, Cristiano

AU - Matos, Miguel

AU - Payer, Mathias

AU - Pirelli, Solal

AU - Portokalidis, Georgios

AU - Schiavoni, Valerio

AU - Signorello, Salvatore

AU - Vahldiek-Oberwagner, Anjo

PY - 2025

Y1 - 2025

N2 - Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.

AB - Artifact Evaluation (“AE”) is now an accepted practice in the systems community. However, AE processes are inconsistent across venues and even across different editions of the same venue. AE processes regularly encounter the same problems across venues and years. Based on our collective experience in chairing various and heterogeneous AE committees for five consecutive editions of EuroSys, a large systems conference, we present the challenges we believe most pressing. We propose concrete steps to address these challenges in future AEs, serving as guidelines for future chairs and AE committees.

KW - Artifact Evaluation

KW - Conference-scale artifact evaluation experiences

KW - Conference-scale artifact evaluation practices

KW - Reproducibility

UR - https://www.scopus.com/pages/publications/105022192359

UR - https://www.scopus.com/inward/citedby.url?scp=105022192359&partnerID=8YFLogxK

U2 - 10.1145/3736731.3746152

DO - 10.1145/3736731.3746152

M3 - Conference contribution

AN - SCOPUS:105022192359

SP - 108

EP - 120

BT - ACM REP '25: Proceedings of the 3rd ACM Conference on Reproducibility and Replicability

PB - Association for Computing Machinery, Inc

T2 - 3rd ACM Conference on Reproducibility and Replicability, ACM REP 2025

Y2 - 29 July 2025 through 31 July 2025

ER -

Lessons Learned from Five Years of Artifact Evaluations at EuroSys

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Cite this