SGD Noise and Implicit Low-Rank Bias in Deep Neural Networks
Name
CBMM-Memo-134.pdf
Size
803.64 KB
Format
Adobe PDF
Checksum (MD5)
73e3e384468f7072cff0d93298181111
Author(s)
Galanti, Tomer
Poggio, Tomaso
Date Issued
March 28, 2022
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo;134
Abstract
We analyze deep ReLU neural networks trained with mini-batch stochastic gradient decent and weight decay. We prove that the source of the SGD noise is an implicit low rank constraint across all of the weight matrices within the network. Furthermore, we show, both theoretically and empirically, that when training a neural network using Stochastic Gradient Descent (SGD) with a small batch size, the resulting weight matrices are expected to be of small rank. Our analysis relies on a minimal set of assumptions and the neural networks may include convolutional layers, residual connections, as well as batch normalization layers.
Persistent DSpace Link
