2026-09-19T17:15:47Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1475612023-01-20T03:34:11Zcom_1721.1_7582com_1721.1_7581col_1721.1_131022
Kaelbling, Leslie P.
Lozano-Pérez, Tomás
Tenenbaum, Joshua B.
Alet i Puig, Ferran
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2023-01-19T19:58:40Z
2023-01-19T19:58:40Z
2022-09
2022-10-19T19:06:36.727Z
https://hdl.handle.net/1721.1/147561
Many recent achievements in machine learning have followed different variations on a single recipe: we pick a supervised training dataset and assume there exists a function mapping inputs to outputs. We then leverage the expressivity of deep learning (together with few but carefully chosen inductive biases for each domain) and train a neural network to approximate this unknown function. In this thesis, we show that this single-function, single-neural-network approach can be too constraining and instead suggest spawning per-point models. This allows us to encode inductive biases in flexible ways and model expressive, structured generative models of the data distribution.
First, we present Tailoring: a novel way of encoding inductive biases by optimizing unsupervised objectives inside the prediction function. This ensures the structure is imposed both at training and test time. Furthermore, its generality allows applications in domains as diverse as physics time-series prediction, adversarial defenses, and contrastive representation learning. We also propose Noether Networks, which automatically discover these inductive biases, in the form of conservation laws.
Finally, we propose Functional risk minimization(FRM), an alternative framework to the standard Empirical risk minimization(ERM) setting where loss functions act in function space rather than output space. We show how we can make learning in this new framework efficient and can lead to improved performance compared to the standard ML setting.
Ph.D.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
Learning inside the prediction function
Thesis
application/pdf
Doctoral
Doctor of Philosophy
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="7cd301e7-0c9f-46ec-badc-aecba5fab42f">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Learning inside the prediction function</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2022-09</PublicationDate>
<Authors>
<Author>
<DisplayName>Alet i Puig, Ferran</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Many recent achievements in machine learning have followed different variations on a single recipe: we pick a supervised training dataset and assume there exists a function mapping inputs to outputs. We then leverage the expressivity of deep learning (together with few but carefully chosen inductive biases for each domain) and train a neural network to approximate this unknown function. In this thesis, we show that this single-function, single-neural-network approach can be too constraining and instead suggest spawning per-point models. This allows us to encode inductive biases in flexible ways and model expressive, structured generative models of the data distribution.
First, we present Tailoring: a novel way of encoding inductive biases by optimizing unsupervised objectives inside the prediction function. This ensures the structure is imposed both at training and test time. Furthermore, its generality allows applications in domains as diverse as physics time-series prediction, adversarial defenses, and contrastive representation learning. We also propose Noether Networks, which automatically discover these inductive biases, in the form of conservation laws.
Finally, we propose Functional risk minimization(FRM), an alternative framework to the standard Empirical risk minimization(ERM) setting where loss functions act in function space rather than output space. We show how we can make learning in this new framework efficient and can lead to improved performance compared to the standard ML setting.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>