Dynamic speculative optimizations for SQL compilation in apache spark

Filippo Schiavio; Daniele Bonetta; Walter Binder

doi:10.14778/3377369.3377382

Dynamic speculative optimizations for SQL compilation in apache spark

Filippo Schiavio
, Daniele Bonetta
, Walter Binder

Research output: Contribution to Journal › Article › Academic › peer-review

Abstract

Big-data systems have gained significant momentum, and Apache Spark is becoming a de-facto standard for modern data analytics. Spark relies on SQL query compilation to optimize the execution performance of analytical workloads on a variety of data sources. Despite its scalable architecture, Spark's SQL code generation suers from significant runtime overheads related to data access and de-serialization. Such performance penalty can be significant, especially when applications operate on human-readable data formats such as CSV or JSON. In this paper we present a new approach to query compilation that overcomes these limitations by relying on runtime profiling and dynamic code generation. Our new SQL compiler for Spark produces highly-efficient machine code, leading to speedups of up to 4.4x on the TPC-H benchmark with textual-form data formats such as CSV or JSON.

Original language	English
Pages (from-to)	754-767
Journal	Proceedings of the VLDB Endowment
Volume	13
Issue number	5
DOIs	https://doi.org/10.14778/3377369.3377382
Publication status	Published - 2020
Externally published	Yes

Funding

The work presented in this paper has been supported by Oracle (ERO project 1332) and the Swiss National Science Foundation (project 200020 188688). We thank the VLDB reviewers for their detailed feedback and the VM Research Group at Oracle Labs for their support. Oracle, Java, and HotSpot are trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.

Funders	Funder number
VM Research Group at Oracle Labs
Schweizerischer Nationalfonds zur Förderung der Wissenschaftlichen Forschung	200020 188688

Access to Document

10.14778/3377369.3377382

Persistent URL (handle)

Persistent link

Cite this

@article{4cc957ad9c3749588044788e05e6fb4b,

title = "Dynamic speculative optimizations for SQL compilation in apache spark",

abstract = "Big-data systems have gained significant momentum, and Apache Spark is becoming a de-facto standard for modern data analytics. Spark relies on SQL query compilation to optimize the execution performance of analytical workloads on a variety of data sources. Despite its scalable architecture, Spark's SQL code generation suers from significant runtime overheads related to data access and de-serialization. Such performance penalty can be significant, especially when applications operate on human-readable data formats such as CSV or JSON. In this paper we present a new approach to query compilation that overcomes these limitations by relying on runtime profiling and dynamic code generation. Our new SQL compiler for Spark produces highly-efficient machine code, leading to speedups of up to 4.4x on the TPC-H benchmark with textual-form data formats such as CSV or JSON.",

author = "Filippo Schiavio and Daniele Bonetta and Walter Binder",

year = "2020",

doi = "10.14778/3377369.3377382",

language = "English",

volume = "13",

pages = "754--767",

journal = "Proceedings of the VLDB Endowment",

issn = "2150-8097",

publisher = "VLDB Endowment",

number = "5",

}

TY - JOUR

T1 - Dynamic speculative optimizations for SQL compilation in apache spark

AU - Schiavio, Filippo

AU - Bonetta, Daniele

AU - Binder, Walter

PY - 2020

Y1 - 2020

N2 - Big-data systems have gained significant momentum, and Apache Spark is becoming a de-facto standard for modern data analytics. Spark relies on SQL query compilation to optimize the execution performance of analytical workloads on a variety of data sources. Despite its scalable architecture, Spark's SQL code generation suers from significant runtime overheads related to data access and de-serialization. Such performance penalty can be significant, especially when applications operate on human-readable data formats such as CSV or JSON. In this paper we present a new approach to query compilation that overcomes these limitations by relying on runtime profiling and dynamic code generation. Our new SQL compiler for Spark produces highly-efficient machine code, leading to speedups of up to 4.4x on the TPC-H benchmark with textual-form data formats such as CSV or JSON.

AB - Big-data systems have gained significant momentum, and Apache Spark is becoming a de-facto standard for modern data analytics. Spark relies on SQL query compilation to optimize the execution performance of analytical workloads on a variety of data sources. Despite its scalable architecture, Spark's SQL code generation suers from significant runtime overheads related to data access and de-serialization. Such performance penalty can be significant, especially when applications operate on human-readable data formats such as CSV or JSON. In this paper we present a new approach to query compilation that overcomes these limitations by relying on runtime profiling and dynamic code generation. Our new SQL compiler for Spark produces highly-efficient machine code, leading to speedups of up to 4.4x on the TPC-H benchmark with textual-form data formats such as CSV or JSON.

UR - https://www.scopus.com/pages/publications/85089939615

U2 - 10.14778/3377369.3377382

DO - 10.14778/3377369.3377382

M3 - Article

SN - 2150-8097

VL - 13

SP - 754

EP - 767

JO - Proceedings of the VLDB Endowment

JF - Proceedings of the VLDB Endowment

IS - 5

ER -

Dynamic speculative optimizations for SQL compilation in apache spark

Abstract

Funding

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this