VectorH: Taking SQL-on-Hadoop to the next level

Andrei Costea; Adrian Ionescu; Bogdan Raducanu; Michał Świtakowski; Cristian Bârca; Juliusz Sompolski; Alicja Łuszczak; Michał Szafrański; Giel De Nijs; Peter Boncz

doi:10.1145/2882903.2903742

VectorH: Taking SQL-on-Hadoop to the next level

Andrei Costea, Adrian Ionescu, Bogdan Raducanu, Michał Świtakowski, Cristian Bârca, Juliusz Sompolski, Alicja Łuszczak, Michał Szafrański, Giel De Nijs, Peter Boncz

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

Abstract

Actian Vector in Hadoop (VectorH for short) is a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and storage scalability by relying on HDFS, and extends the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS replication policy to optimize read locality. VectorH integrates with YARN for workload management, achieving a high degree of elasticity. Even though HDFS is an append-only file-system, and VectorH supports (update-averse) ordered tables, trickle updates are possible thanks to Positional Delta Trees (PDTs), a diffferential update structure that can be queried efficiently. We describe the changes made to single-server Vectorwise to turn it into a Hadoop-based MPP system, encompassing workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. We evaluate VectorH against HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance.

Original language	English
Title of host publication	SIGMOD 2016 - Proceedings of the 2016 International Conference on Management of Data
Publisher	Association for Computing Machinery (ACM)
Pages	1105-1117
Number of pages	13
Volume	26-June-2016
ISBN (Electronic)	9781450335317
DOIs	https://doi.org/10.1145/2882903.2903742
Publication status	Published - 26 Jun 2016
Event	2016 ACM SIGMOD International Conference on Management of Data, SIGMOD 2016 - San Francisco, United States Duration: 26 Jun 2016 → 1 Jul 2016

Conference

Conference	2016 ACM SIGMOD International Conference on Management of Data, SIGMOD 2016
Country/Territory	United States
City	San Francisco
Period	26/06/16 → 1/07/16

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1145/2882903.2903742

Persistent URL (handle)

Copy link address

Cite this

Costea, A., Ionescu, A., Raducanu, B., Świtakowski, M., Bârca, C., Sompolski, J., Łuszczak, A., Szafrański, M., De Nijs, G., & Boncz, P. (2016). VectorH: Taking SQL-on-Hadoop to the next level. In SIGMOD 2016 - Proceedings of the 2016 International Conference on Management of Data (Vol. 26-June-2016, pp. 1105-1117). Association for Computing Machinery (ACM). https://doi.org/10.1145/2882903.2903742

@inproceedings{81f07f72c43d466a8c80eb41d581cc13,

title = "VectorH: Taking SQL-on-Hadoop to the next level",

abstract = "Actian Vector in Hadoop (VectorH for short) is a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and storage scalability by relying on HDFS, and extends the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS replication policy to optimize read locality. VectorH integrates with YARN for workload management, achieving a high degree of elasticity. Even though HDFS is an append-only file-system, and VectorH supports (update-averse) ordered tables, trickle updates are possible thanks to Positional Delta Trees (PDTs), a diffferential update structure that can be queried efficiently. We describe the changes made to single-server Vectorwise to turn it into a Hadoop-based MPP system, encompassing workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. We evaluate VectorH against HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance.",

author = "Andrei Costea and Adrian Ionescu and Bogdan Raducanu and Micha{\l} {\'S}witakowski and Cristian B{\^a}rca and Juliusz Sompolski and Alicja {\L}uszczak and Micha{\l} Szafra{\'n}ski and {De Nijs}, Giel and Peter Boncz",

year = "2016",

month = jun,

day = "26",

doi = "10.1145/2882903.2903742",

language = "English",

volume = "26-June-2016",

pages = "1105--1117",

booktitle = "SIGMOD 2016 - Proceedings of the 2016 International Conference on Management of Data",

publisher = "Association for Computing Machinery (ACM)",

note = "2016 ACM SIGMOD International Conference on Management of Data, SIGMOD 2016 ; Conference date: 26-06-2016 Through 01-07-2016",

}

Costea, A, Ionescu, A, Raducanu, B, Świtakowski, M, Bârca, C, Sompolski, J, Łuszczak, A, Szafrański, M, De Nijs, G & Boncz, P 2016, VectorH: Taking SQL-on-Hadoop to the next level. in SIGMOD 2016 - Proceedings of the 2016 International Conference on Management of Data. vol. 26-June-2016, Association for Computing Machinery (ACM), pp. 1105-1117, 2016 ACM SIGMOD International Conference on Management of Data, SIGMOD 2016, San Francisco, United States, 26/06/16. https://doi.org/10.1145/2882903.2903742

VectorH: Taking SQL-on-Hadoop to the next level. / Costea, Andrei; Ionescu, Adrian; Raducanu, Bogdan et al.
SIGMOD 2016 - Proceedings of the 2016 International Conference on Management of Data. Vol. 26-June-2016 Association for Computing Machinery (ACM), 2016. p. 1105-1117.

Research output: Chapter in Book / Report / Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - VectorH

T2 - 2016 ACM SIGMOD International Conference on Management of Data, SIGMOD 2016

AU - Costea, Andrei

AU - Ionescu, Adrian

AU - Raducanu, Bogdan

AU - Świtakowski, Michał

AU - Bârca, Cristian

AU - Sompolski, Juliusz

AU - Łuszczak, Alicja

AU - Szafrański, Michał

AU - De Nijs, Giel

AU - Boncz, Peter

PY - 2016/6/26

Y1 - 2016/6/26

N2 - Actian Vector in Hadoop (VectorH for short) is a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and storage scalability by relying on HDFS, and extends the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS replication policy to optimize read locality. VectorH integrates with YARN for workload management, achieving a high degree of elasticity. Even though HDFS is an append-only file-system, and VectorH supports (update-averse) ordered tables, trickle updates are possible thanks to Positional Delta Trees (PDTs), a diffferential update structure that can be queried efficiently. We describe the changes made to single-server Vectorwise to turn it into a Hadoop-based MPP system, encompassing workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. We evaluate VectorH against HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance.

AB - Actian Vector in Hadoop (VectorH for short) is a new SQL-on-Hadoop system built on top of the fast Vectorwise analytical database system. VectorH achieves fault tolerance and storage scalability by relying on HDFS, and extends the state-of-the-art in SQL-on-Hadoop systems by instrumenting the HDFS replication policy to optimize read locality. VectorH integrates with YARN for workload management, achieving a high degree of elasticity. Even though HDFS is an append-only file-system, and VectorH supports (update-averse) ordered tables, trickle updates are possible thanks to Positional Delta Trees (PDTs), a diffferential update structure that can be queried efficiently. We describe the changes made to single-server Vectorwise to turn it into a Hadoop-based MPP system, encompassing workload management, parallel query optimization and execution, HDFS storage, transaction processing and Spark integration. We evaluate VectorH against HAWQ, Impala, SparkSQL and Hive, showing orders of magnitude better performance.

UR - http://www.scopus.com/inward/record.url?scp=84979710919&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84979710919&partnerID=8YFLogxK

U2 - 10.1145/2882903.2903742

DO - 10.1145/2882903.2903742

M3 - Conference contribution

AN - SCOPUS:84979710919

VL - 26-June-2016

SP - 1105

EP - 1117

BT - SIGMOD 2016 - Proceedings of the 2016 International Conference on Management of Data

PB - Association for Computing Machinery (ACM)

Y2 - 26 June 2016 through 1 July 2016

ER -

VectorH: Taking SQL-on-Hadoop to the next level

Abstract

Conference

UN SDGs

Access to Document

Persistent URL (handle)

Other files and links

Fingerprint

Cite this