BayesDB : querying the probable implications of tabular data

Abstract

BayesDB, a Bayesian database table, lets users query the probable implications of their tabular data as easily as an SQL database lets them query the data itself. Using the built-in Bayesian Query Language (BQL), users with little statistics knowledge can solve basic data science problems, such as detecting predictive relationships between variables, inferring missing values, simulating probable observations, and identifying statistically similar database entries. BayesDB is suitable for analyzing complex, heterogeneous data tables with no preprocessing or parameter adjustment required. This generality rests on the model independence provided by BQL, analogous to the physical data independence provided by the relational model. SQL enables data filtering and aggregation tasks to be described independently of the physical layout of data in memory and on disk. Non-experts rely on generic indexing strategies for good-enough performance, while experts customize schemes and indices for performance-sensitive applications. Analogously, BQL enables analysis tasks to be described independently of the models used to solve them. Non-statisticians can rely on a general-purpose modeling method called CrossCat to build models that are good enough for a broad class of applications, while experts can customize the schemes and models when needed. This thesis defines BQL, describes an implementation of BayesDB, quantitatively characterizes its scalability and performance, and illustrates its efficacy on real-world data analysis problems in the areas of healthcare economics, statistical survey data analysis, web analytics, and predictive policing.