Deep Learning without Poor Local Minima
Name
MIT-CSAIL-TR-2016-005.pdf
Size
336.43 KB
Format
Adobe PDF
Checksum (MD5)
a7a9d05913f0443995300a6405ea7cbc
Download all files submitted through automated deposit
Author(s)
Kawaguchi, Kenji
Advisor(s)
Leslie Kaelbling
Date Issued
May 23, 2016
Series/Report no.
MIT-CSAIL-TR-2016-005
Abstract
In this paper, we prove a conjecture published in 1989 and also partially address an open problem announced at the Conference on Learning Theory (COLT) 2015. For an expected loss function of a deep nonlinear neural network, we prove the following statements under the independence assumption adopted from recent work: 1) the function is non-convex and non-concave, 2) every local minimum is a global minimum, 3) every critical point that is not a global minimum is a saddle point, and 4) the property of saddle points differs for shallow networks (with three layers) and deeper networks (with more than three layers). Moreover, we prove that the same four statements hold for deep linear neural networks with any depth, any widths and no unrealistic assumptions. As a result, we present an instance, for which we can answer to the following question: how difficult to directly train a deep model in theory? It is more difficult than the classical machine learning models (because of the non-convexity), but not too difficult (because of the nonexistence of poor local minima and the property of the saddle points). We note that even though we have advanced the theoretical foundations of deep learning, there is still a gap between theory and practice.
Subjects
Optimization
Neural Network
Machine Learning
High Dimension
Convex
Non-convex
Local minimum
Global minimum
Saddle point
Critical point
Terms of Use
Creative Commons Attribution 4.0 International
Persistent DSpace Link
