Yardstick: A benchmark for minecraft-like services

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Online gaming applications entertain hundreds of millions of daily active players and often feature vastly complex architecture. Among online games, Minecraft-like games simulate unique (e.g., modifiable) environments, are virally popular, and are increasingly provided as a service. However, the performance of Minecraft-like services, and in particular their scalability, is not well understood. Moreover, currently no benchmark exists for Minecraft-like games. Addressing this knowledge gap, in this work we design and use the Yardstick benchmark to analyze the performance of Minecraft-like services. Yardstick is based on an operational model that captures salient characteristics of Minecraft-like services. As input workload, Yardstick captures important features, such as the most-popular maps used within the Minecraft community. Yardstick captures system- and application-level metrics, and derives from them service-level metrics such as frequency of game-updates under scalable workload. We implement Yardstick, and, through real-world experiments in our clusters, we explore the performance and scalability of popular Minecraft-like servers, including the official vanilla server, and the community-developed servers Spigot and Glowstone. Our findings indicate the scalability limits of these servers, that Minecraft-like services are poorly parallelized, and that Glowstone is the least viable option among those tested.

Original languageEnglish
Title of host publicationICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering
Place of PublicationNew York, NY
PublisherAssociation for Computing Machinery, Inc
Pages242-252
Number of pages11
ISBN (Electronic)9781450362399
DOIs
Publication statusPublished - 4 Apr 2019
Event10th ACM/SPEC International Conference on Performance Engineering, ICPE 2019 - Mumbai, India
Duration: 7 Apr 201911 Apr 2019

Conference

Conference10th ACM/SPEC International Conference on Performance Engineering, ICPE 2019
CountryIndia
CityMumbai
Period7/04/1911/04/19

Fingerprint

Servers
Scalability
Experiments

Keywords

  • As a service
  • Benchmark
  • Distributed systems
  • Minecraft
  • Online gaming
  • Yardstick

Cite this

Van Der Sar, J., Donkervliet, J., & Iosup, A. (2019). Yardstick: A benchmark for minecraft-like services. In ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering (pp. 242-252). New York, NY: Association for Computing Machinery, Inc. https://doi.org/10.1145/3297663.3310307
Van Der Sar, Jerom ; Donkervliet, Jesse ; Iosup, Alexandru. / Yardstick : A benchmark for minecraft-like services. ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. New York, NY : Association for Computing Machinery, Inc, 2019. pp. 242-252
@inproceedings{40670dc7defc41bfa4f86a020fd4b219,
title = "Yardstick: A benchmark for minecraft-like services",
abstract = "Online gaming applications entertain hundreds of millions of daily active players and often feature vastly complex architecture. Among online games, Minecraft-like games simulate unique (e.g., modifiable) environments, are virally popular, and are increasingly provided as a service. However, the performance of Minecraft-like services, and in particular their scalability, is not well understood. Moreover, currently no benchmark exists for Minecraft-like games. Addressing this knowledge gap, in this work we design and use the Yardstick benchmark to analyze the performance of Minecraft-like services. Yardstick is based on an operational model that captures salient characteristics of Minecraft-like services. As input workload, Yardstick captures important features, such as the most-popular maps used within the Minecraft community. Yardstick captures system- and application-level metrics, and derives from them service-level metrics such as frequency of game-updates under scalable workload. We implement Yardstick, and, through real-world experiments in our clusters, we explore the performance and scalability of popular Minecraft-like servers, including the official vanilla server, and the community-developed servers Spigot and Glowstone. Our findings indicate the scalability limits of these servers, that Minecraft-like services are poorly parallelized, and that Glowstone is the least viable option among those tested.",
keywords = "As a service, Benchmark, Distributed systems, Minecraft, Online gaming, Yardstick",
author = "{Van Der Sar}, Jerom and Jesse Donkervliet and Alexandru Iosup",
year = "2019",
month = "4",
day = "4",
doi = "10.1145/3297663.3310307",
language = "English",
pages = "242--252",
booktitle = "ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering",
publisher = "Association for Computing Machinery, Inc",

}

Van Der Sar, J, Donkervliet, J & Iosup, A 2019, Yardstick: A benchmark for minecraft-like services. in ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. Association for Computing Machinery, Inc, New York, NY, pp. 242-252, 10th ACM/SPEC International Conference on Performance Engineering, ICPE 2019, Mumbai, India, 7/04/19. https://doi.org/10.1145/3297663.3310307

Yardstick : A benchmark for minecraft-like services. / Van Der Sar, Jerom; Donkervliet, Jesse; Iosup, Alexandru.

ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. New York, NY : Association for Computing Machinery, Inc, 2019. p. 242-252.

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Yardstick

T2 - A benchmark for minecraft-like services

AU - Van Der Sar, Jerom

AU - Donkervliet, Jesse

AU - Iosup, Alexandru

PY - 2019/4/4

Y1 - 2019/4/4

N2 - Online gaming applications entertain hundreds of millions of daily active players and often feature vastly complex architecture. Among online games, Minecraft-like games simulate unique (e.g., modifiable) environments, are virally popular, and are increasingly provided as a service. However, the performance of Minecraft-like services, and in particular their scalability, is not well understood. Moreover, currently no benchmark exists for Minecraft-like games. Addressing this knowledge gap, in this work we design and use the Yardstick benchmark to analyze the performance of Minecraft-like services. Yardstick is based on an operational model that captures salient characteristics of Minecraft-like services. As input workload, Yardstick captures important features, such as the most-popular maps used within the Minecraft community. Yardstick captures system- and application-level metrics, and derives from them service-level metrics such as frequency of game-updates under scalable workload. We implement Yardstick, and, through real-world experiments in our clusters, we explore the performance and scalability of popular Minecraft-like servers, including the official vanilla server, and the community-developed servers Spigot and Glowstone. Our findings indicate the scalability limits of these servers, that Minecraft-like services are poorly parallelized, and that Glowstone is the least viable option among those tested.

AB - Online gaming applications entertain hundreds of millions of daily active players and often feature vastly complex architecture. Among online games, Minecraft-like games simulate unique (e.g., modifiable) environments, are virally popular, and are increasingly provided as a service. However, the performance of Minecraft-like services, and in particular their scalability, is not well understood. Moreover, currently no benchmark exists for Minecraft-like games. Addressing this knowledge gap, in this work we design and use the Yardstick benchmark to analyze the performance of Minecraft-like services. Yardstick is based on an operational model that captures salient characteristics of Minecraft-like services. As input workload, Yardstick captures important features, such as the most-popular maps used within the Minecraft community. Yardstick captures system- and application-level metrics, and derives from them service-level metrics such as frequency of game-updates under scalable workload. We implement Yardstick, and, through real-world experiments in our clusters, we explore the performance and scalability of popular Minecraft-like servers, including the official vanilla server, and the community-developed servers Spigot and Glowstone. Our findings indicate the scalability limits of these servers, that Minecraft-like services are poorly parallelized, and that Glowstone is the least viable option among those tested.

KW - As a service

KW - Benchmark

KW - Distributed systems

KW - Minecraft

KW - Online gaming

KW - Yardstick

UR - http://www.scopus.com/inward/record.url?scp=85064811726&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064811726&partnerID=8YFLogxK

U2 - 10.1145/3297663.3310307

DO - 10.1145/3297663.3310307

M3 - Conference contribution

SP - 242

EP - 252

BT - ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering

PB - Association for Computing Machinery, Inc

CY - New York, NY

ER -

Van Der Sar J, Donkervliet J, Iosup A. Yardstick: A benchmark for minecraft-like services. In ICPE 2019 - Proceedings of the 2019 ACM/SPEC International Conference on Performance Engineering. New York, NY: Association for Computing Machinery, Inc. 2019. p. 242-252 https://doi.org/10.1145/3297663.3310307