Towards resource disaggregation - Memory scavenging for scientific workloads

Alexandru Uta, Ana Maria Oprescu, T. Kielmann

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Compute clusters, consisting of many, uniformly built nodes, are used to run a large spectrum of different workloads, like tightly coupled (MPI) jobs, MapReduce, or graph-processing data-analytics applications, each of which with their own resource requirements. Many studies consistently highlight two types of under-utilized cluster resources: memory (up to 50%) and network. In this work, we take a step towards (software) resource disaggregation, and therefore increased resource utilization, by designing a memory scavenging technique that makes unused memory available to applications on other cluster nodes. We implement this technique in MemFSS, an inmemory distributed file system. The scavenging MemFSS extends its storage space by taking advantage of the unused memory and bandwidth of cluster nodes already running other tenants' applications. Our experiments show that our memory scavenging approach incurs negligible overhead (below 10%) for most tenant applications, while the compute resource comsumption of MemFSS applications is largely reduced (by 17%-74%).

Original languageEnglish
Title of host publicationProceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016
PublisherInstitute of Electrical and Electronics Engineers, Inc.
Pages100-109
Number of pages10
ISBN (Electronic)9781509036530
DOIs
Publication statusPublished - 6 Dec 2016
Event2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 - Taipei, Taiwan, Province of China
Duration: 13 Sep 201615 Sep 2016

Conference

Conference2016 IEEE International Conference on Cluster Computing, CLUSTER 2016
CountryTaiwan, Province of China
CityTaipei
Period13/09/1615/09/16

Fingerprint

Scavenging
Data storage equipment
Bandwidth
Experiments

Cite this

Uta, A., Oprescu, A. M., & Kielmann, T. (2016). Towards resource disaggregation - Memory scavenging for scientific workloads. In Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 (pp. 100-109). [7776483] Institute of Electrical and Electronics Engineers, Inc.. https://doi.org/10.1109/CLUSTER.2016.18
Uta, Alexandru ; Oprescu, Ana Maria ; Kielmann, T. / Towards resource disaggregation - Memory scavenging for scientific workloads. Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016. Institute of Electrical and Electronics Engineers, Inc., 2016. pp. 100-109
@inproceedings{51a9a1ed812d44b09164f41555d5243e,
title = "Towards resource disaggregation - Memory scavenging for scientific workloads",
abstract = "Compute clusters, consisting of many, uniformly built nodes, are used to run a large spectrum of different workloads, like tightly coupled (MPI) jobs, MapReduce, or graph-processing data-analytics applications, each of which with their own resource requirements. Many studies consistently highlight two types of under-utilized cluster resources: memory (up to 50{\%}) and network. In this work, we take a step towards (software) resource disaggregation, and therefore increased resource utilization, by designing a memory scavenging technique that makes unused memory available to applications on other cluster nodes. We implement this technique in MemFSS, an inmemory distributed file system. The scavenging MemFSS extends its storage space by taking advantage of the unused memory and bandwidth of cluster nodes already running other tenants' applications. Our experiments show that our memory scavenging approach incurs negligible overhead (below 10{\%}) for most tenant applications, while the compute resource comsumption of MemFSS applications is largely reduced (by 17{\%}-74{\%}).",
author = "Alexandru Uta and Oprescu, {Ana Maria} and T. Kielmann",
year = "2016",
month = "12",
day = "6",
doi = "10.1109/CLUSTER.2016.18",
language = "English",
pages = "100--109",
booktitle = "Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016",
publisher = "Institute of Electrical and Electronics Engineers, Inc.",

}

Uta, A, Oprescu, AM & Kielmann, T 2016, Towards resource disaggregation - Memory scavenging for scientific workloads. in Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016., 7776483, Institute of Electrical and Electronics Engineers, Inc., pp. 100-109, 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016, Taipei, Taiwan, Province of China, 13/09/16. https://doi.org/10.1109/CLUSTER.2016.18

Towards resource disaggregation - Memory scavenging for scientific workloads. / Uta, Alexandru; Oprescu, Ana Maria; Kielmann, T.

Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016. Institute of Electrical and Electronics Engineers, Inc., 2016. p. 100-109 7776483.

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

TY - GEN

T1 - Towards resource disaggregation - Memory scavenging for scientific workloads

AU - Uta, Alexandru

AU - Oprescu, Ana Maria

AU - Kielmann, T.

PY - 2016/12/6

Y1 - 2016/12/6

N2 - Compute clusters, consisting of many, uniformly built nodes, are used to run a large spectrum of different workloads, like tightly coupled (MPI) jobs, MapReduce, or graph-processing data-analytics applications, each of which with their own resource requirements. Many studies consistently highlight two types of under-utilized cluster resources: memory (up to 50%) and network. In this work, we take a step towards (software) resource disaggregation, and therefore increased resource utilization, by designing a memory scavenging technique that makes unused memory available to applications on other cluster nodes. We implement this technique in MemFSS, an inmemory distributed file system. The scavenging MemFSS extends its storage space by taking advantage of the unused memory and bandwidth of cluster nodes already running other tenants' applications. Our experiments show that our memory scavenging approach incurs negligible overhead (below 10%) for most tenant applications, while the compute resource comsumption of MemFSS applications is largely reduced (by 17%-74%).

AB - Compute clusters, consisting of many, uniformly built nodes, are used to run a large spectrum of different workloads, like tightly coupled (MPI) jobs, MapReduce, or graph-processing data-analytics applications, each of which with their own resource requirements. Many studies consistently highlight two types of under-utilized cluster resources: memory (up to 50%) and network. In this work, we take a step towards (software) resource disaggregation, and therefore increased resource utilization, by designing a memory scavenging technique that makes unused memory available to applications on other cluster nodes. We implement this technique in MemFSS, an inmemory distributed file system. The scavenging MemFSS extends its storage space by taking advantage of the unused memory and bandwidth of cluster nodes already running other tenants' applications. Our experiments show that our memory scavenging approach incurs negligible overhead (below 10%) for most tenant applications, while the compute resource comsumption of MemFSS applications is largely reduced (by 17%-74%).

UR - http://www.scopus.com/inward/record.url?scp=85013140439&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85013140439&partnerID=8YFLogxK

U2 - 10.1109/CLUSTER.2016.18

DO - 10.1109/CLUSTER.2016.18

M3 - Conference contribution

SP - 100

EP - 109

BT - Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016

PB - Institute of Electrical and Electronics Engineers, Inc.

ER -

Uta A, Oprescu AM, Kielmann T. Towards resource disaggregation - Memory scavenging for scientific workloads. In Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016. Institute of Electrical and Electronics Engineers, Inc. 2016. p. 100-109. 7776483 https://doi.org/10.1109/CLUSTER.2016.18