Beneficial Initializations in Over-Parameterized Machine Learning Problems
Name
Prasad-nehap-meng-eecs-2021-thesis.pdf
Description
Thesis PDF
Size
1.9 MB
Format
Adobe PDF
Checksum (MD5)
dd82869a2310c61d84b5e3d55a047484
Author(s)
Prasad, Neha
Advisor(s)
Uhler, Caroline
Date Issued
June 2021
Publisher
Massachusetts Institute of Technology
Abstract
We theoretically and empirically analyze the phenomenon of transfer learning in overparameterized machine learning. We start by showing that in over-parameterized linear regression, transfer learning is equivalent to solving regression from a non-zero initialization. We use this finding to propose LLBoost, a theoretically grounded, computationally efficient method to boost the validation and test accuracy of pretrained, over-parameterized models without impacting the original training accuracy. We evaluate LLBoost on CIFAR10, ImageNet-32, and ImageNet and also prove that it reduces the generalization error of any interpolating solution with high probability. By extending our analysis of transfer learning in linear regression, we present an approach for transfer learning in kernel regression. Namely, we demonstrate that transfer learning corresponds to adding a function to the minimum norm solution that produces zero error on the training data. We use this approach to perform transfer learning on image classification tasks using the neural tangent kernel.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
