Virtualizing Cloud Data Infrastructures with BRAD

Abstract

Organizations usually manage their data using multiple specialized cloud database engines (e.g., Aurora, BigQuery, etc.). However, designing and managing multi-engine infrastructures is hard; there can be many designs, each with different performance and costs. Changing the design afterwards (e.g., due to growth) is even more challenging since application code usually ends up tightly coupled to the engines. We propose data infrastructure virtualization. The key idea is to declare a set of virtual database engines (VDBEs), which specify an engine's application-facing properties (e.g., query interface, performance) and its tables, but do not prescribe a concrete engine. An automated planner then decides how to best realize the VDBEs onto physical engines based on the workload. Clients connect to VDBE endpoints and are oblivious to the underlying physical engines-allowing for seamless infrastructure changes. We implemented VDBEs and an automated planner in BRAD: the first data infrastructure virtualization runtime. Our demo will showcase VDBEs and BRAD's automated planner under different workloads.