Bayesian tuning and bandits : an extensible, open source library for AutoML

Abstract

The goal of this thesis is to build an extensible and open source library that handles the problems of tuning the hyperparameters of a machine learning pipeline, selecting between multiple pipelines, and recommending a pipeline. We devise a library that users can integrate into their existing datascience workflows and experts can contribute to by writing methods to solve these search problems. Extending upon the existing library, our goals are twofold: one that the library naturally fits within a user's existing workflow, so that integration does not require a lot of overhead, and two that the three search problems are broken down into small and modular pieces to allow contributors to have maximal flexibility. We establish the abstractions for each of the solutions to these search problems, showcasing how both a user would use the library and a contributor could override the API. We discuss the creation of a recommender system, that proposes machine learning pipelines for a new dataset, trained on an existing matrix of known scores of pipelines on datasets. We show how using such a system can lead to performance gains. We discuss how we can evaluate the quality of different solutions to these types of search problems, and how we can measurably compare them to each other.