Bayesian Active Structure Learning for Gaussian Process Probabilistic Programs

Abstract

What data should we gather to learn about the underlying structure of the world as quickly as possible, especially in cases where data is sparse or expensive to acquire? Structure learning techniques for Gaussian process (GP) probabilistic programs provide a rich framework for inferring qualitative structure in data. In this thesis, we improve the data-efficiency of probabilistic GP structure learning by extending it to the active learning setting. We present a sequential Monte Carlo algorithm for Bayesian active learning for GPs with a novel objective function, Kernel Information Gain (IG-K), to reduce uncertainty over model structure and parameters. As a baseline for comparison, we also formulate a second objective function, Predictive Information Gain (IG-P), that reduces uncertainty over the posterior predictive distribution.

We empirically validate that active learning with our novel IG-K objective is able to more accurately infer the structure of synthetic datasets using fewer datapoints than active learning with IG-P. We also validate the underlying active learning inference algorithm using simulation-based calibration. Finally, we test our active learning algorithm on a real-world dataset with complex structure. Collectively, the results provide a deeper understanding of the benefits and limitations of active structure learning using Gaussian processes, revealing that an active selection strategy suited for inferring the model structure and parameters may not favorable for providing accurate predictions. These findings suggest directions for future active learning approaches which combine the IG-K and IG-P objectives, leveraging the advantages of each objective to efficiently discover structure in data and provide accurate predictions.