A reference architecture for datacenter scheduling: Design, validation, and experiments

Georgios Andreadis, Laurens Versluis, Fabian Mastenbroek, Alexandru Iosup

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Datacenters act as cloud-infrastructure to stakeholders across industry, government, and academia. To meet growing demand yet operate efficiently, datacenter operators employ increasingly more sophisticated scheduling systems, mechanisms, and policies. Although many scheduling techniques already exist, relatively little research has gone into the abstraction of the scheduling process itself, hampering design, tuning, and comparison of existing techniques. In this work, we propose a reference architecture for datacenter schedulers. The architecture follows five design principles: components with clearly distinct responsibilities, grouping of related components where possible, separation of mechanism from policy, scheduling as complex workflow, and hierarchical multi-scheduler structure. To demonstrate the validity of the reference architecture, we map to it state-of-the-art datacenter schedulers. We find scheduler-stages are commonly underspecified in peer-reviewed publications. Through trace-based simulation and real-world experiments, we show underspecification of scheduler-stages can lead to significant variations in performance.

Original languageEnglish
Title of host publicationProceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages478-492
Number of pages15
ISBN (Electronic)9781538683842
DOIs
Publication statusPublished - 11 Mar 2019
Event2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 - Dallas, United States
Duration: 11 Nov 201816 Nov 2018

Conference

Conference2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
CountryUnited States
CityDallas
Period11/11/1816/11/18

Fingerprint

Scheduler
Scheduling
Experiment
Experiments
Scheduling Policy
Tuning
Grouping
Work Flow
Infrastructure
Trace
Design
Architecture
Industry
Distinct
Operator
Demonstrate
Simulation

Keywords

  • Datacenter
  • Reference Architecture
  • Scheduling

Cite this

Andreadis, G., Versluis, L., Mastenbroek, F., & Iosup, A. (2019). A reference architecture for datacenter scheduling: Design, validation, and experiments. In Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 (pp. 478-492). [8665816] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SC.2018.00040
Andreadis, Georgios ; Versluis, Laurens ; Mastenbroek, Fabian ; Iosup, Alexandru. / A reference architecture for datacenter scheduling : Design, validation, and experiments. Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 478-492
@inproceedings{d70232c1fa384981b7bd1b02ce44c773,
title = "A reference architecture for datacenter scheduling: Design, validation, and experiments",
abstract = "Datacenters act as cloud-infrastructure to stakeholders across industry, government, and academia. To meet growing demand yet operate efficiently, datacenter operators employ increasingly more sophisticated scheduling systems, mechanisms, and policies. Although many scheduling techniques already exist, relatively little research has gone into the abstraction of the scheduling process itself, hampering design, tuning, and comparison of existing techniques. In this work, we propose a reference architecture for datacenter schedulers. The architecture follows five design principles: components with clearly distinct responsibilities, grouping of related components where possible, separation of mechanism from policy, scheduling as complex workflow, and hierarchical multi-scheduler structure. To demonstrate the validity of the reference architecture, we map to it state-of-the-art datacenter schedulers. We find scheduler-stages are commonly underspecified in peer-reviewed publications. Through trace-based simulation and real-world experiments, we show underspecification of scheduler-stages can lead to significant variations in performance.",
keywords = "Datacenter, Reference Architecture, Scheduling",
author = "Georgios Andreadis and Laurens Versluis and Fabian Mastenbroek and Alexandru Iosup",
year = "2019",
month = "3",
day = "11",
doi = "10.1109/SC.2018.00040",
language = "English",
pages = "478--492",
booktitle = "Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

Andreadis, G, Versluis, L, Mastenbroek, F & Iosup, A 2019, A reference architecture for datacenter scheduling: Design, validation, and experiments. in Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018., 8665816, Institute of Electrical and Electronics Engineers Inc., pp. 478-492, 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, United States, 11/11/18. https://doi.org/10.1109/SC.2018.00040

A reference architecture for datacenter scheduling : Design, validation, and experiments. / Andreadis, Georgios; Versluis, Laurens; Mastenbroek, Fabian; Iosup, Alexandru.

Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc., 2019. p. 478-492 8665816.

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - A reference architecture for datacenter scheduling

T2 - Design, validation, and experiments

AU - Andreadis, Georgios

AU - Versluis, Laurens

AU - Mastenbroek, Fabian

AU - Iosup, Alexandru

PY - 2019/3/11

Y1 - 2019/3/11

N2 - Datacenters act as cloud-infrastructure to stakeholders across industry, government, and academia. To meet growing demand yet operate efficiently, datacenter operators employ increasingly more sophisticated scheduling systems, mechanisms, and policies. Although many scheduling techniques already exist, relatively little research has gone into the abstraction of the scheduling process itself, hampering design, tuning, and comparison of existing techniques. In this work, we propose a reference architecture for datacenter schedulers. The architecture follows five design principles: components with clearly distinct responsibilities, grouping of related components where possible, separation of mechanism from policy, scheduling as complex workflow, and hierarchical multi-scheduler structure. To demonstrate the validity of the reference architecture, we map to it state-of-the-art datacenter schedulers. We find scheduler-stages are commonly underspecified in peer-reviewed publications. Through trace-based simulation and real-world experiments, we show underspecification of scheduler-stages can lead to significant variations in performance.

AB - Datacenters act as cloud-infrastructure to stakeholders across industry, government, and academia. To meet growing demand yet operate efficiently, datacenter operators employ increasingly more sophisticated scheduling systems, mechanisms, and policies. Although many scheduling techniques already exist, relatively little research has gone into the abstraction of the scheduling process itself, hampering design, tuning, and comparison of existing techniques. In this work, we propose a reference architecture for datacenter schedulers. The architecture follows five design principles: components with clearly distinct responsibilities, grouping of related components where possible, separation of mechanism from policy, scheduling as complex workflow, and hierarchical multi-scheduler structure. To demonstrate the validity of the reference architecture, we map to it state-of-the-art datacenter schedulers. We find scheduler-stages are commonly underspecified in peer-reviewed publications. Through trace-based simulation and real-world experiments, we show underspecification of scheduler-stages can lead to significant variations in performance.

KW - Datacenter

KW - Reference Architecture

KW - Scheduling

UR - http://www.scopus.com/inward/record.url?scp=85064117463&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85064117463&partnerID=8YFLogxK

U2 - 10.1109/SC.2018.00040

DO - 10.1109/SC.2018.00040

M3 - Conference contribution

SP - 478

EP - 492

BT - Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Andreadis G, Versluis L, Mastenbroek F, Iosup A. A reference architecture for datacenter scheduling: Design, validation, and experiments. In Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc. 2019. p. 478-492. 8665816 https://doi.org/10.1109/SC.2018.00040