Theory of Deep Learning III: explaining the non-overfitting puzzle
Name
CBMM-Memo-073.pdf
Size
2.65 MB
Format
Adobe PDF
Checksum (MD5)
bd0716aadb2959efe4df8164d0ae3332
Author(s)
Poggio, Tomaso
Kawaguchi, Kenji
Liao, Qianli
Miranda, Brando
Rosasco, Lorenzo
Boix, Xavier
Hidary, Jack
Mhaskar, Hrushikesh
Date Issued
December 30, 2017
Publisher
arXiv
Citation
arXiv:1801.00173
Series/Report no.
CBMM Memo Series;073
Abstract
THIS MEMO IS REPLACED BY CBMM MEMO 90
A main puzzle of deep networks revolves around the absence of overfitting despite overparametrization and despite the large capacity demonstrated by zero training error on randomly labeled data. In this note, we show that the dynamical systems associated with gradient descent minimization of nonlinear networks behave near zero stable minima of the empirical error as gradient system in a quadratic potential with degenerate Hessian. The proposition is supported by theoretical and numerical results, under the assumption of stable minima of the gradient.
Our proposition provides the extension to deep networks of key properties of gradient descent methods for linear networks, that as, suggested in (1), can be the key to understand generalization. Gradient descent enforces a form of implicit regular- ization controlled by the number of iterations, and asymptotically converging to the minimum norm solution. This implies that there is usually an optimum early stopping that avoids overfitting of the loss (this is relevant mainly for regression). For classification, the asymptotic convergence to the minimum norm solution implies convergence to the maximum margin solution which guarantees good classification error for “low noise” datasets.
The implied robustness to overparametrization has suggestive implications for the robustness of deep hierarchically local networks to variations of the architecture with respect to the curse of dimensionality.
Terms of Use
Attribution-NonCommercial-ShareAlike 3.0 United States
Persistent DSpace Link
