Visualizations for model tracking and predictions in machine learning

Abstract

Building machine learning models is often an exploratory and iterative process. A data scientist frequently builds and trains hundreds of models with different parameters and feature sets in order to find one that meets the desired criteria. However, it can be difficult to keep track of all the parameters and metadata that are associated with the models. ModelDB, an end-to-end system for managing machine learning models, is a tool that solves this problem of model management. In this thesis, we present a graphical user interface for ModelDB, along with an extension for visualizing model predictions. The core user interface for model management augments the ModelDB system, which previously consisted only of native client libraries and a backend. The interface provides new ways of exploring, visualizing, and analyzing model data through a web application. The prediction visualizations extend the core user interface by providing a novel prediction matrix that displays classifier outputs in order to convey model performance at the example level. We present the design and implementation of both the core user interface and the prediction visualizations, discussing at each step the motivations behind key features. We evaluate the prediction visualizations through a pilot user study, which produces preliminary feedback on the practicality and utility of the interface. The overall goal of this research is to provide a powerful, user-friendly interface that leverages the data stored in ModelDB to generate effective visualizations for analyzing and improving models.