Beneficial Initializations in Over-Parameterized Machine Learning Problems

Abstract

We theoretically and empirically analyze the phenomenon of transfer learning in overparameterized machine learning. We start by showing that in over-parameterized linear regression, transfer learning is equivalent to solving regression from a non-zero initialization. We use this finding to propose LLBoost, a theoretically grounded, computationally efficient method to boost the validation and test accuracy of pretrained, over-parameterized models without impacting the original training accuracy. We evaluate LLBoost on CIFAR10, ImageNet-32, and ImageNet and also prove that it reduces the generalization error of any interpolating solution with high probability. By extending our analysis of transfer learning in linear regression, we present an approach for transfer learning in kernel regression. Namely, we demonstrate that transfer learning corresponds to adding a function to the minimum norm solution that produces zero error on the training data. We use this approach to perform transfer learning on image classification tasks using the neural tangent kernel.