Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. MIT Open Access Articles
  3. MIT Open Access Articles
  4. Vista: Machine Learning based Database Performance Troubleshooting Framework in Amazon RDS

Vista: Machine Learning based Database Performance Troubleshooting Framework in Amazon RDS

Thumbnail Image
Download
Name

3698038.3698519.pdf

Size

14.49 MB

Format

Adobe PDF

Checksum (MD5)

1588a150703dab354b951bff9846e7e9

Download all files submitted through automated deposit
3698038.3698519.zip (10.16 MB)
Author(s)
Singh, Vikramank
•
Song, Zhao
•
Narayanaswamy, Balakrishnan (Murali)
•
Vaidya, Kapil Eknath
•
Kraska, Tim
Date Issued
November 20, 2024
Publisher
ACM|ACM Symposium on Cloud Computing
Citation
Singh, Vikramank, Song, Zhao, Narayanaswamy, Balakrishnan (Murali), Vaidya, Kapil Eknath and Kraska, Tim. 2024. "Vista: Machine Learning based Database Performance Troubleshooting Framework in Amazon RDS."
Version
Final published version
Abstract
Database performance troubleshooting is a complex multi-step process that broadly involves three key stages- (a) Detection: determining what's wrong and when; (b) Root Cause Analysis (RCA): reasoning about why is the performance poor; (c) Resolution: identifying a fix. A plethora of techniques exist to address each of these problems, but they hardly work in real-world at scale. First, real-world customer workloads are noisy, non-stationary and quasi-periodic in nature rendering traditional detectors ineffective. Second, real-world production databases execute a highly diverse set of queries that skew the database statistics into long-tail distributions causing traditional RCA methods to fail. Third, these databases typically execute millions of such diverse queries every minute rendering traditional methods inefficient when deployed at scale. In this paper we describe Vista, a machine learning based performance troubleshooting framework for databases, and dive-deep into how it addresses the 3 real-world problems outlined above. Vista deploys a deep auto-regressive model trained on a large and diverse Amazon Relational Database Service (RDS) fleet with custom skip connections and periodicity alignment features to model long range and varying periodicity in customer workloads, and detects performance bottlenecks in the form of outliers. Furthermore, it efficiently filters only a top few dominating SQL queries from millions in a problematic workload, and uses a robust causal inference framework to identify the culprit queries and their statistics leading to a low false-positive and false-negative rate. Currently, Vista runs on hundreds of thousands of RDS databases, analyzes millions of workloads every day bringing down the troubleshooting time for RDS customers from hours to seconds. At the end, we also describe several challenges and learnings from implementing and deploying Vista at Amazon scale.
Description
SoCC ’24, November 20–22, 2024, Redmond, WA, USA
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution
https://creativecommons.org/licenses/by/4.0/
Persistent DSpace Link
https://hdl.handle.net/1721.1/157897
DOI of Published Version
https://doi.org/10.1145/3698038.3698519
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.