Learning inside the prediction function
Name
Alet-alet-PhD-EECS-2022-thesis.pdf
Description
Thesis PDF
Size
18.16 MB
Format
Adobe PDF
Checksum (MD5)
a26fc80677dba44abdc17dd90297bbec
Author(s)
Alet i Puig, Ferran
Advisor(s)
Kaelbling, Leslie P.
Lozano-Pérez, Tomás
Tenenbaum, Joshua B.
Date Issued
September 2022
Publisher
Massachusetts Institute of Technology
Abstract
Many recent achievements in machine learning have followed different variations on a single recipe: we pick a supervised training dataset and assume there exists a function mapping inputs to outputs. We then leverage the expressivity of deep learning (together with few but carefully chosen inductive biases for each domain) and train a neural network to approximate this unknown function. In this thesis, we show that this single-function, single-neural-network approach can be too constraining and instead suggest spawning per-point models. This allows us to encode inductive biases in flexible ways and model expressive, structured generative models of the data distribution.
First, we present Tailoring: a novel way of encoding inductive biases by optimizing unsupervised objectives inside the prediction function. This ensures the structure is imposed both at training and test time. Furthermore, its generality allows applications in domains as diverse as physics time-series prediction, adversarial defenses, and contrastive representation learning. We also propose Noether Networks, which automatically discover these inductive biases, in the form of conservation laws.
Finally, we propose Functional risk minimization(FRM), an alternative framework to the standard Empirical risk minimization(ERM) setting where loss functions act in function space rather than output space. We show how we can make learning in this new framework efficient and can lead to improved performance compared to the standard ML setting.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
