Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

Jerrit Eickhoff; Jesse Donkervliet; Alexandru Iosup

doi:10.1109/ISPASS55109.2022.00018

Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

Jerrit Eickhoff
, Jesse Donkervliet
, Alexandru Iosup

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

393 Downloads (Pure)

Abstract

One of the most popular types of online games is the Minecraft-like Game (MLG), in which players can terraform the environment. MLGs currently support their many players by replicating isolated instances with limited scalability. We posit that performance variability is a key cause for the lack of scalability in MLGs and design the first benchmark that focuses on MLG performance variability, identifying specialized workloads, metrics, and processes. We conduct real-world benchmarking of MLGs, both cloud-based and self-hosted. We find environment-based workloads and cloud deployment are significant sources of performance variability: peak-latency degrades sharply to 20.7 times the arithmetic mean, and exceeds by a factor of 7.4 the performance requirements.

Original language	English
Title of host publication	IEEE International Symposium on Performance Analysis of Systems and Software
Subtitle of host publication	[Proceedings]
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	147-149
Number of pages	3
ISBN (Electronic)	9781665459549
ISBN (Print)	9781665459556
DOIs	https://doi.org/10.1109/ISPASS55109.2022.00018
Publication status	Published - 2022
Event	2022 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2022 - Singapore, Singapore Duration: 22 May 2022 → 24 May 2022

Conference

Conference	2022 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2022
Country/Territory	Singapore
City	Singapore
Period	22/05/22 → 24/05/22

Bibliographical note

Publisher Copyright:
© 2022 IEEE.

Keywords

Benchmarking
Online Games
Performance Variability
Workloads

Access to Document

10.1109/ISPASS55109.2022.00018

MeterstickFinal published version, 613 KBLicence: Article 25fa Dutch Copyright Act

Persistent URL (handle)

Persistent link

Cite this

@inproceedings{6b06c20a00a5486289e505f36c316cd5,

title = "Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games",

abstract = "One of the most popular types of online games is the Minecraft-like Game (MLG), in which players can terraform the environment. MLGs currently support their many players by replicating isolated instances with limited scalability. We posit that performance variability is a key cause for the lack of scalability in MLGs and design the first benchmark that focuses on MLG performance variability, identifying specialized workloads, metrics, and processes. We conduct real-world benchmarking of MLGs, both cloud-based and self-hosted. We find environment-based workloads and cloud deployment are significant sources of performance variability: peak-latency degrades sharply to 20.7 times the arithmetic mean, and exceeds by a factor of 7.4 the performance requirements. ",

keywords = "Benchmarking, Online Games, Performance Variability, Workloads",

author = "Jerrit Eickhoff and Jesse Donkervliet and Alexandru Iosup",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 2022 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2022 ; Conference date: 22-05-2022 Through 24-05-2022",

year = "2022",

doi = "10.1109/ISPASS55109.2022.00018",

language = "English",

isbn = "9781665459556",

pages = "147--149",

booktitle = "IEEE International Symposium on Performance Analysis of Systems and Software",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

}

Eickhoff, J, Donkervliet, J & Iosup, A 2022, Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games. in IEEE International Symposium on Performance Analysis of Systems and Software: [Proceedings]. Institute of Electrical and Electronics Engineers Inc., pp. 147-149, 2022 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2022, Singapore, Singapore, 22/05/22. https://doi.org/10.1109/ISPASS55109.2022.00018

Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games. / Eickhoff, Jerrit; Donkervliet, Jesse; Iosup, Alexandru.
IEEE International Symposium on Performance Analysis of Systems and Software: [Proceedings]. Institute of Electrical and Electronics Engineers Inc., 2022. p. 147-149.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

AU - Eickhoff, Jerrit

AU - Donkervliet, Jesse

AU - Iosup, Alexandru

PY - 2022

Y1 - 2022

N2 - One of the most popular types of online games is the Minecraft-like Game (MLG), in which players can terraform the environment. MLGs currently support their many players by replicating isolated instances with limited scalability. We posit that performance variability is a key cause for the lack of scalability in MLGs and design the first benchmark that focuses on MLG performance variability, identifying specialized workloads, metrics, and processes. We conduct real-world benchmarking of MLGs, both cloud-based and self-hosted. We find environment-based workloads and cloud deployment are significant sources of performance variability: peak-latency degrades sharply to 20.7 times the arithmetic mean, and exceeds by a factor of 7.4 the performance requirements.

AB - One of the most popular types of online games is the Minecraft-like Game (MLG), in which players can terraform the environment. MLGs currently support their many players by replicating isolated instances with limited scalability. We posit that performance variability is a key cause for the lack of scalability in MLGs and design the first benchmark that focuses on MLG performance variability, identifying specialized workloads, metrics, and processes. We conduct real-world benchmarking of MLGs, both cloud-based and self-hosted. We find environment-based workloads and cloud deployment are significant sources of performance variability: peak-latency degrades sharply to 20.7 times the arithmetic mean, and exceeds by a factor of 7.4 the performance requirements.

KW - Benchmarking

KW - Online Games

KW - Performance Variability

KW - Workloads

UR - https://www.scopus.com/pages/publications/85134304930

UR - https://www.scopus.com/pages/publications/85134304930#tab=citedBy

U2 - 10.1109/ISPASS55109.2022.00018

DO - 10.1109/ISPASS55109.2022.00018

M3 - Conference contribution

AN - SCOPUS:85134304930

SN - 9781665459556

SP - 147

EP - 149

BT - IEEE International Symposium on Performance Analysis of Systems and Software

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2022 IEEE International Symposium on Performance Analysis of Systems and Software, ISPASS 2022

Y2 - 22 May 2022 through 24 May 2022

ER -

Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

Abstract

Conference

Bibliographical note

Keywords

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Meterstick: Benchmarking Performance Variability in Cloud and Self-hosted Minecraft-like Games

Cite this