Albis: High-performance file format for big data systems

Animesh Trivedi, Patrick Stuedi, Jonas Pfefferle, Adrian Schuepbach, Bernard Metzler

Research output: Chapter in Book / Report / Conference proceedingConference contributionAcademicpeer-review

Abstract

Over the last decade, a variety of external file formats such as Parquet, ORC, Arrow, etc., have been developed to store large volumes of relational data in the cloud. As high-performance networking and storage devices are used pervasively to process this data in frameworks like Spark and Hadoop, we observe that none of the popular file formats are capable of delivering data access rates close to the hardware. Our analysis suggests that multiple antiquated notions about the nature of I/O in a distributed setting, and the preference for the “storage efficiency” over performance is the key reason for this gap. In this paper we present Albis, a high-performance file format for storing relational data on modern hardware. Albis is built upon two key principles: (i) reduce the CPU cost by keeping the data/metadata storage format simple; (ii) use a binary API for an efficient object management to avoid unnecessary object materialization. In our evaluation, we demonstrate that in micro-benchmarks Albis delivers 1.9 − 21.4× faster bandwidths than other formats. At the workload-level, Albis in Spark/SQL reduces the runtimes of TPC-DS queries up to a margin of 3×.

Original languageEnglish
Title of host publicationProceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018
PublisherUSENIX Association
Pages615-629
Number of pages15
ISBN (Electronic)9781939133021
Publication statusPublished - 1 Jan 2020
Externally publishedYes
Event2018 USENIX Annual Technical Conference, USENIX ATC 2018 - Boston, United States
Duration: 11 Jul 201813 Jul 2018

Publication series

NameProceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018

Conference

Conference2018 USENIX Annual Technical Conference, USENIX ATC 2018
CountryUnited States
CityBoston
Period11/07/1813/07/18

    Fingerprint

Cite this

Trivedi, A., Stuedi, P., Pfefferle, J., Schuepbach, A., & Metzler, B. (2020). Albis: High-performance file format for big data systems. In Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018 (pp. 615-629). (Proceedings of the 2018 USENIX Annual Technical Conference, USENIX ATC 2018). USENIX Association.