Theoretical issues in deep networks
Name
30039.full.pdf
Description
Published version
Size
911.25 KB
Format
Adobe PDF
Checksum (MD5)
1cd33502d2a6bfc1536e10e47448ebe5
Author(s)
Poggio, Tomaso
Banburski, Andrzej
Liao, Qianli
Date Issued
2020
Journal
Proceedings of the National Academy of Sciences of the United States of America
Publisher
Proceedings of the National Academy of Sciences
Version
Final published version
Abstract
© 2020 National Academy of Sciences. All rights reserved. While deep learning is successful in a number of applications, it is not yet well understood theoretically. A theoretical characterization of deep learning should answer questions about their approximation power, the dynamics of optimization, and good out-of-sample performance, despite overparameterization and the absence of explicit regularization. We review our recent results toward this goal. In approximation theory both shallow and deep networks are known to approximate any continuous functions at an exponential cost. However, we proved that for certain types of compositional functions, deep networks of the convolutional type (even without weight sharing) can avoid the curse of dimensionality. In characterizing minimization of the empirical exponential loss we consider the gradient flow of the weight directions rather than the weights themselves, since the relevant function underlying classification corresponds to normalized networks. The dynamics of normalized weights turn out to be equivalent to those of the constrained problem of minimizing the loss subject to a unit norm constraint. In particular, the dynamics of typical gradient descent have the same critical points as the constrained problem. Thus there is implicit regularization in training deep networks under exponential-type loss functions during gradient flow. As a consequence, the critical points correspond to minimum norm infima of the loss. This result is especially relevant because it has been recently shown that, for overparameterized models, selection of a minimum norm solution optimizes cross-validation leave-one-out stability and thereby the expected error. Thus our results imply that gradient descent in deep networks minimize the expected error.
MIT Department
Center for Brains, Minds, and Machines
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1073/PNAS.1907369117
