A robust partitioning scheme for ad-hoc query workloads
Name
hyperPartitioning.pdf
Description
Accepted version
Size
3.11 MB
Format
Adobe PDF
Checksum (MD5)
8c019f9362c347a7f626b83938e909c6
Author(s)
Shanbhag, Anil
Jindal, Alekh
Madden, Samuel
Quiane, Jorge
Elmore, Aaron J.
Date Issued
September 24, 2017
Publisher
ACM
Citation
Shanbhag, Anil, Jindal, Alekh, Madden, Samuel, Quiane, Jorge and Elmore, Aaron J. 2017. "A robust partitioning scheme for ad-hoc query workloads."
Version
Author's final manuscript
Abstract
© 2017 Association for Computing Machinery. Data partitioning is crucial to improving query performance and severalworkload-based partitioning techniques have been proposed in database literature. However, many modern analytic applications involve ad-hoc or exploratory analysis where users do not have a representative query workload a priori. Static workload-based data partitioning techniques are therefore not suitable for such settings. In this paper, we propose Amoeba, a distributed storage system that uses adaptive multi-attribute data partitioning to efficiently support ad-hoc as well as recurring queries. Amoeba requires zero set-up and tuning effort, allowing analysts to get the benefits of partitioning without requiring an upfront query workload. The key idea is to build and maintain a partitioning tree on top of the dataset. The partitioning tree allows us to answer queries with predicates by reading a subset of the data. The initial partitioning tree is created without requiring an upfront query workload and Amoeba adapts it over time by incrementally modifying subtrees based on user queries using repartitioning. A prototype of Amoeba running on top of Apache Spark improves query performance by up to 7x over full scans and up to 2x over range-based partitioning techniques on TPC-H as well as a real-world workload.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3127479.3131613
