Beyond microbenchmarks: The SPEC-RG vision for a comprehensive serverless benchmark

Erwin Van Eyk, Joel Scheuner, Simon Eismann, Cristina L. Abad, Alexandru Iosup

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Serverless computing services, such as Function-as-a-Service (FaaS), hold the attractive promise of a high level of abstraction and high performance, combined with the minimization of operational logic. Several large ecosystems of serverless platforms, both open- and closed-source, aim to realize this promise. Consequently, a lucrative market has emerged. However, the performance trade-offs of these systems are not well-understood. Moreover, it is exactly the high level of abstraction and the opaqueness of the operational-side that make performance evaluation studies of serverless platforms challenging. Learning from the history of IT platforms, we argue that a benchmark for serverless platforms could help address this challenge. We envision a comprehensive serverless benchmark, which we contrast to the narrow focus of prior work in this area. We argue that a comprehensive benchmark will need to take into account more than just runtime overhead, and include notions of cost, realistic workloads, more (open-source) platforms, and cloud integrations. Finally, we show through preliminary real-world experiments how such a benchmark can help compare the performance overhead when running a serverless workload on state-of-the-art platforms.

Original languageEnglish
Title of host publicationICPE '20
Subtitle of host publicationCompanion of the ACM/SPEC International Conference on Performance Engineering
PublisherAssociation for Computing Machinery, Inc
Pages26-31
Number of pages6
ISBN (Electronic)9781450371094
DOIs
Publication statusPublished - 20 Apr 2020
Event11th ACM/SPEC International Conference on Performance Engineering, ICPE 2020 - Edmonton, Canada
Duration: 20 Apr 202024 Apr 2020

Conference

Conference11th ACM/SPEC International Conference on Performance Engineering, ICPE 2020
CountryCanada
CityEdmonton
Period20/04/2024/04/20

Keywords

  • function-as-a-service
  • performance
  • serverless computing

Fingerprint Dive into the research topics of 'Beyond microbenchmarks: The SPEC-RG vision for a comprehensive serverless benchmark'. Together they form a unique fingerprint.

Cite this