An elasticity study of distributed graph processing

Sietse Au, Alexandru Uta, Alexey Ilyushkin, Alexandru Iosup

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the variety of graph data and algorithms, many parallel and distributed graph processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore a dynamic model of deployment. We first characterize workload dynamicity, beyond mere active-vertex variability. Then, to conduct an in-depth elasticity study of distributed graph processing, we build a prototype, JoyGraph, which is the first such system that implements complex, policy-based, and fine-grained elasticity. Using the state-of-the-art LDBC Graphalytics benchmark and the SPEC Cloud Group's elasticity metrics, we show the benefits of elasticity in graph processing: (i) improved resource utilization, (ii) reduced operational costs, and (iii) aligned operation-workload dynamicity. Furthermore, we explore the cost of elasticity in graph processing. We identify a key drawback: although elasticity does not degrade application throughput, graph-processing workloads are sensitive to data movement while leasing or releasing resources.

LanguageEnglish
Title of host publicationProceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages382-383
Number of pages2
ISBN (Electronic)9781538658154
DOIs
Publication statusPublished - 13 Jul 2018
Event18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018 - Washington, United States
Duration: 1 May 20184 May 2018

Conference

Conference18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018
CountryUnited States
CityWashington
Period1/05/184/05/18

Fingerprint

Elasticity
Processing
Costs
Dynamic models
Throughput

Keywords

  • Auto sclaing
  • Autoscaling
  • Cost
  • Distributed graph processing
  • Elasticity
  • Graph
  • Graph processing
  • Graphalytics
  • Joygraph
  • SPEC
  • Utilization

Cite this

Au, S., Uta, A., Ilyushkin, A., & Iosup, A. (2018). An elasticity study of distributed graph processing. In Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018 (pp. 382-383). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CCGRID.2018.00062
Au, Sietse ; Uta, Alexandru ; Ilyushkin, Alexey ; Iosup, Alexandru. / An elasticity study of distributed graph processing. Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 382-383
@inproceedings{d9389713c3294920b34f29e4fbdd8c5f,
title = "An elasticity study of distributed graph processing",
abstract = "Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the variety of graph data and algorithms, many parallel and distributed graph processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore a dynamic model of deployment. We first characterize workload dynamicity, beyond mere active-vertex variability. Then, to conduct an in-depth elasticity study of distributed graph processing, we build a prototype, JoyGraph, which is the first such system that implements complex, policy-based, and fine-grained elasticity. Using the state-of-the-art LDBC Graphalytics benchmark and the SPEC Cloud Group's elasticity metrics, we show the benefits of elasticity in graph processing: (i) improved resource utilization, (ii) reduced operational costs, and (iii) aligned operation-workload dynamicity. Furthermore, we explore the cost of elasticity in graph processing. We identify a key drawback: although elasticity does not degrade application throughput, graph-processing workloads are sensitive to data movement while leasing or releasing resources.",
keywords = "Auto sclaing, Autoscaling, Cost, Distributed graph processing, Elasticity, Graph, Graph processing, Graphalytics, Joygraph, SPEC, Utilization",
author = "Sietse Au and Alexandru Uta and Alexey Ilyushkin and Alexandru Iosup",
year = "2018",
month = "7",
day = "13",
doi = "10.1109/CCGRID.2018.00062",
language = "English",
pages = "382--383",
booktitle = "Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018",
publisher = "Institute of Electrical and Electronics Engineers Inc.",
address = "United States",

}

Au, S, Uta, A, Ilyushkin, A & Iosup, A 2018, An elasticity study of distributed graph processing. in Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018. Institute of Electrical and Electronics Engineers Inc., pp. 382-383, 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018, Washington, United States, 1/05/18. https://doi.org/10.1109/CCGRID.2018.00062

An elasticity study of distributed graph processing. / Au, Sietse; Uta, Alexandru; Ilyushkin, Alexey; Iosup, Alexandru.

Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 382-383.

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - An elasticity study of distributed graph processing

AU - Au, Sietse

AU - Uta, Alexandru

AU - Ilyushkin, Alexey

AU - Iosup, Alexandru

PY - 2018/7/13

Y1 - 2018/7/13

N2 - Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the variety of graph data and algorithms, many parallel and distributed graph processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore a dynamic model of deployment. We first characterize workload dynamicity, beyond mere active-vertex variability. Then, to conduct an in-depth elasticity study of distributed graph processing, we build a prototype, JoyGraph, which is the first such system that implements complex, policy-based, and fine-grained elasticity. Using the state-of-the-art LDBC Graphalytics benchmark and the SPEC Cloud Group's elasticity metrics, we show the benefits of elasticity in graph processing: (i) improved resource utilization, (ii) reduced operational costs, and (iii) aligned operation-workload dynamicity. Furthermore, we explore the cost of elasticity in graph processing. We identify a key drawback: although elasticity does not degrade application throughput, graph-processing workloads are sensitive to data movement while leasing or releasing resources.

AB - Graphs are a natural fit for modeling concepts used in solving diverse problems in science, commerce, engineering, and governance. Responding to the variety of graph data and algorithms, many parallel and distributed graph processing systems exist. However, until now these platforms use a static model of deployment: they only run on a pre-defined set of machines. This raises many conceptual and pragmatic issues, including misfit with the highly dynamic nature of graph processing, and could lead to resource waste and high operational costs. In contrast, in this work we explore a dynamic model of deployment. We first characterize workload dynamicity, beyond mere active-vertex variability. Then, to conduct an in-depth elasticity study of distributed graph processing, we build a prototype, JoyGraph, which is the first such system that implements complex, policy-based, and fine-grained elasticity. Using the state-of-the-art LDBC Graphalytics benchmark and the SPEC Cloud Group's elasticity metrics, we show the benefits of elasticity in graph processing: (i) improved resource utilization, (ii) reduced operational costs, and (iii) aligned operation-workload dynamicity. Furthermore, we explore the cost of elasticity in graph processing. We identify a key drawback: although elasticity does not degrade application throughput, graph-processing workloads are sensitive to data movement while leasing or releasing resources.

KW - Auto sclaing

KW - Autoscaling

KW - Cost

KW - Distributed graph processing

KW - Elasticity

KW - Graph

KW - Graph processing

KW - Graphalytics

KW - Joygraph

KW - SPEC

KW - Utilization

UR - http://www.scopus.com/inward/record.url?scp=85050964545&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85050964545&partnerID=8YFLogxK

U2 - 10.1109/CCGRID.2018.00062

DO - 10.1109/CCGRID.2018.00062

M3 - Conference contribution

SP - 382

EP - 383

BT - Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018

PB - Institute of Electrical and Electronics Engineers Inc.

ER -

Au S, Uta A, Ilyushkin A, Iosup A. An elasticity study of distributed graph processing. In Proceedings - 18th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, CCGRID 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 382-383 https://doi.org/10.1109/CCGRID.2018.00062