Search
Now showing items 1-1 of 1
SGD and Weight Decay Provably Induce a Low-Rank Bias in Deep Neural Networks
(Center for Brains, Minds and Machines (CBMM), 2023-02-14)
In this paper, we study the bias of Stochastic Gradient Descent (SGD) to learn low-rank weight matrices when training deep ReLU neural networks. Our results show that training neural networks with mini-batch SGD and weight ...