2026-09-19T11:19:27Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1390502022-01-15T03:11:47Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Uhler, Caroline
Prasad, Neha
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2022-01-14T14:46:51Z
2022-01-14T14:46:51Z
2021-06
2021-06-17T20:14:05.349Z
https://hdl.handle.net/1721.1/139050
We theoretically and empirically analyze the phenomenon of transfer learning in overparameterized machine learning. We start by showing that in over-parameterized linear regression, transfer learning is equivalent to solving regression from a non-zero initialization. We use this finding to propose LLBoost, a theoretically grounded, computationally efficient method to boost the validation and test accuracy of pretrained, over-parameterized models without impacting the original training accuracy. We evaluate LLBoost on CIFAR10, ImageNet-32, and ImageNet and also prove that it reduces the generalization error of any interpolating solution with high probability. By extending our analysis of transfer learning in linear regression, we present an approach for transfer learning in kernel regression. Namely, we demonstrate that transfer learning corresponds to adding a function to the minimum norm solution that produces zero error on the training data. We use this approach to perform transfer learning on image classification tasks using the neural tangent kernel.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
Beneficial Initializations in Over-Parameterized Machine Learning Problems
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="c9253a4e-8446-4ede-8b34-b2c425483635">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Beneficial Initializations in Over-Parameterized Machine Learning Problems</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2021-06</PublicationDate>
<Authors>
<Author>
<DisplayName>Prasad, Neha</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>We theoretically and empirically analyze the phenomenon of transfer learning in overparameterized machine learning. We start by showing that in over-parameterized linear regression, transfer learning is equivalent to solving regression from a non-zero initialization. We use this finding to propose LLBoost, a theoretically grounded, computationally efficient method to boost the validation and test accuracy of pretrained, over-parameterized models without impacting the original training accuracy. We evaluate LLBoost on CIFAR10, ImageNet-32, and ImageNet and also prove that it reduces the generalization error of any interpolating solution with high probability. By extending our analysis of transfer learning in linear regression, we present an approach for transfer learning in kernel regression. Namely, we demonstrate that transfer learning corresponds to adding a function to the minimum norm solution that produces zero error on the training data. We use this approach to perform transfer learning on image classification tasks using the neural tangent kernel.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>