Learning inside the prediction function

Abstract

Many recent achievements in machine learning have followed different variations on a single recipe: we pick a supervised training dataset and assume there exists a function mapping inputs to outputs. We then leverage the expressivity of deep learning (together with few but carefully chosen inductive biases for each domain) and train a neural network to approximate this unknown function. In this thesis, we show that this single-function, single-neural-network approach can be too constraining and instead suggest spawning per-point models. This allows us to encode inductive biases in flexible ways and model expressive, structured generative models of the data distribution.

First, we present Tailoring: a novel way of encoding inductive biases by optimizing unsupervised objectives inside the prediction function. This ensures the structure is imposed both at training and test time. Furthermore, its generality allows applications in domains as diverse as physics time-series prediction, adversarial defenses, and contrastive representation learning. We also propose Noether Networks, which automatically discover these inductive biases, in the form of conservation laws.

Finally, we propose Functional risk minimization(FRM), an alternative framework to the standard Empirical risk minimization(ERM) setting where loss functions act in function space rather than output space. We show how we can make learning in this new framework efficient and can lead to improved performance compared to the standard ML setting.