Towards an automatic predictive question formulation

Abstract

In this thesis, we designed a formal language, called Trane, for describing prediction problems over relational datasets, implemented a system that allows humans to specify problems in that language, and allows them to build models that solve them using real data. We show that this language is able to describe all 54 prediction problems on the Kaggle data science competition website[14] and so is comprehensive. The implemented system consists of a web application connected to a server-side interpreter, which translates input from the web application into a series of transformation and aggregation operations to apply to a dataset in order to generate labels that can be used to train a supervised machine learning classifier. Using a smaller subset of this language, we developed software that enumerated 1077 prediction problems automatically for the Walmart Store Sales Forecasting dataset found on Kaggle[16], and built models that attempted to solve them, for which we produced 235 AUC scores. The web application also allowed us to collect 157 ratings from humans on the meaningfulness of randomly-generated prediction problems. We used these ratings along with an enumeration of 6105 prediction problems and 7 datasets to train a collaborative-filtering based recommendation system to propose meaningful prediction problems on new, unseen datasets.