SGD and Weight Decay Provably Induce a Low-Rank Bias in Deep Neural Networks
Name
CBMM-Memo-140.pdf
Size
2.38 MB
Format
Adobe PDF
Checksum (MD5)
338ff3db499fff4f0fed7386d086db6e
Author(s)
Galanti, Tomer
Siegel, Zachary
Gupte, Aparna
Poggio, Tomaso
Date Issued
February 14, 2023
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;140
Abstract
In this paper, we study the bias of Stochastic Gradient Descent (SGD) to learn low-rank weight matrices when training deep ReLU neural networks. Our results show that training neural networks with mini-batch SGD and weight decay causes a bias towards rank minimization over the weight matri- ces. Specifically, we show, both theoretically and empirically, that this bias is more pronounced when using smaller batch sizes, higher learning rates, or increased weight decay. Additionally, we predict and observe empirically that weight decay is necessary to achieve this bias. Finally, we empirically investigate the connection between this bias and generalization, finding that it has a marginal effect on generalization. Our analysis is based on a minimal set of assumptions and applies to neural networks of any width or depth, including those with residual connections and convolutional layers.
Persistent DSpace Link
