Instance-Optimized Database Indexes and Storage Layouts

Abstract

For any modern database system, its physical design, which is composed of both the storage layout of the data itself and auxiliary data structures such as indexes, is a critical piece of maintaining high performance in the face of increasing data volumes. Existing physical design components are general-purpose: they achieve adequate performance for the average use case but don’t achieve optimal performance for any individual use case. These physical design components expose numerous configuration knobs that users must manually tune to achieve better performance for their individual use case, but tuning complex systems is labor-intensive, and poor tuning can result in degraded performance and increased costs.

In this thesis, we explore how database systems can maximize performance while minimizing manual effort through instance-optimization, which is the process of designing systems that are able to automatically self-adjust in order to achieve the best performance for a given use case. We leverage instance-optimization to introduce novel designs for database indexes and data storage layouts that outperform existing state-of-the-art indexes and data layouts by orders of magnitude. We also demonstrate how to incorporate multiple instance-optimized database components into an end-toend analytic database system that outperforms a well-tuned commercial cloud-based analytics system by up to 3×.