Defio: Instance-Optimized Fusion of AWS Database Services

Abstract

Building large-scale data infrastructures is hard: There are often more than a single type of workloads and business requirements, but unfortunately, “one size does not fit all”. Modern database systems tend to specialize towards a specific type of workload, and thus organizations are left to integrate these differently-specialized database systems in order to achieve sufficient performance for all of their use cases and workloads.

This kind of hybrid architecture—also known as Data Mesh architecture—often leads to the increasing complexity of maintaining and utilizing database services, both for the data engineers and the end users. However, we believe that some of this complexity can be abstracted away from the end users, in particular with respect to query routing, i.e. determining where to execute each individual SQL query among the multiple database engines.

To overcome this challenge, we propose Defio, a unified interface to multiple specialized database engines that can intelligently handle myriads of workloads without having the end users think about the underlying execution of each query. Specifically, this thesis focuses on the design and implementation of an instance-optimized query router, which ultimately enables Defio to take advantage of the performance benefits of each specialized database in a Data Mesh architecture—resulting in what we call a fusion of database services.