Unification of temporary storage in the nodekernel architecture

Patrick Stuedi, Animesh Trivedi, Jonas Pfefferle, Ana Klimovic, Adrian Schuepbach, Bernard Metzler

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Efficiently exchanging temporary data between tasks is critical to the end-to-end performance of many data processing frameworks and applications. Unfortunately, the diverse nature of temporary data creates storage demands that often fall between the sweet spots of traditional storage platforms, such as file systems or key-value stores. We present NodeKernel, a novel distributed storage architecture that offers a convenient new point in the design space by fusing file system and key-value semantics in a common storage kernel while leveraging modern networking and storage hardware to achieve high performance and cost-efficiency. NodeKernel provides hierarchical naming, high scalability, and close to bare-metal performance for a wide range of data sizes and access patterns that are characteristic of temporary data. We show that storing temporary data in Crail, our concrete implementation of the NodeKernel architecture which uses RDMA networking with tiered DRAM/NVMe-Flash storage, improves NoSQL workload performance by up to 4.8× and Spark application performance by up to 3.4×. Furthermore, by storing data across NVMe Flash and DRAM storage tiers, Crail reduces storage cost by up to 8× compared to DRAM-only storage systems.

Original languageEnglish
Title of host publicationProceedings of the 2019 USENIX Annual Technical Conference, USENIX ATC 2019
PublisherUSENIX Association
Pages767-781
Number of pages15
ISBN (Electronic)9781939133038
Publication statusPublished - 2019
Event2019 USENIX Annual Technical Conference, USENIX ATC 2019 - Renton, United States
Duration: 10 Jul 201912 Jul 2019

Conference

Conference2019 USENIX Annual Technical Conference, USENIX ATC 2019
CountryUnited States
CityRenton
Period10/07/1912/07/19

Fingerprint Dive into the research topics of 'Unification of temporary storage in the nodekernel architecture'. Together they form a unique fingerprint.

Cite this