Balanced resource allocations across multiple dynamic MapReduce clusters

Bogdan Ghit, Nezih Yigitbasi, Alexandru Iosup, Dick H.J. Epema

Research output: Contribution to JournalArticleAcademicpeer-review


Running multiple instances of the MapReduce framework concurrently in a multicluster system or datacenter enables data, failure, and version isolation, which is attractive for many organizations. It may also provide some form of performance isolation, but in order to achieve this in the face of time-varying workloads submitted to the MapReduce instances, a mechanism for dynamic resource (re-)allocations to those instances is required. In this paper, we present such a mechanism called FAWKES that attempts to balance the allocations to MapReduce instances so that they experience similar service levels. FAWKES proposes a new abstraction for deploying MapReduce instances on physical resources, the MR-cluster, which represents a set of resources that can grow and shrink, and that has a core on which MapReduce is installed with the usual data locality assumptions but that relaxes those assumptions for nodes outside the core. FAWKES dynamically grows and shrinks the active MR-clusters based on a family of weighting policies with weights derived from monitoring their operation. We empirically evaluate FAWKES on a multicluster system and show that it can deliver good performance and balanced resource allocations, even when the workloads of the MR-clusters are very uneven and bursty, with workloads composed from both synthetic and real-world benchmarks.
Original languageEnglish
Pages (from-to)329-341
Number of pages13
JournalACM Sigmetrics Performance Evaluation Review
Issue number1
Publication statusPublished - 20 Jun 2014
Externally publishedYes
Event2014 ACM SIGMETRICS International Conference on Measurement and Modeling of Computer Systems, SIGMETRICS 2014 - Austin, United States
Duration: 16 Jun 201420 Jun 2014


  • Datacenters
  • Dynamic provisioning
  • Fairness
  • Mapreduce clusters
  • Multicluster systems
  • Performance
  • Scheduling


Dive into the research topics of 'Balanced resource allocations across multiple dynamic MapReduce clusters'. Together they form a unique fingerprint.

Cite this