Every Local Minimum Value Is the Global Minimum Value of Induced Model in Nonconvex Machine Learning
Name
neco_a_01234.pdf
Description
Published version
Size
556.97 KB
Format
Adobe PDF
Checksum (MD5)
de3cf8702fada45e7306e3c418c01998
Author(s)
Kawaguchi, Kenji
Huang, Jiaoyang
Kaelbling, Leslie P
Date Issued
December 2019
Journal
Neural Computation
Publisher
MIT Press - Journals
Citation
Kawaguchi, Kenji et al. "Every Local Minimum Value Is the Global Minimum Value of Induced Model in Nonconvex Machine Learning." Neural Computation 31, 12 (December 2019): 2293-2323 © 2019 Massachusetts Institute of Technology
Version
Final published version
Abstract
For nonconvex optimization in machine learning, this article proves that every local minimum achieves the globally optimal value of the perturbable gradient basis model at any differentiable point. As a result, nonconvex machine learning is theoretically as supported as convex machine learning with a handcrafted basis in terms of the loss at differentiable local minima, except in the case when a preference is given to the handcrafted basis over the perturbable gradient basis. The proofs of these results are derived under mild assumptions. Accordingly, the proven results are directly applicable to many machine learning models, including practical deep neural networks, without any modification of practical methods. Furthermore, as special cases of our general results, this article improves or complements several state-of-the-art theoretical results on deep neural networks, deep residual networks, and overparameterized deep neural networks with a unified proof technique and novel geometric insights. A special case of our results also contributes to the theoretical foundation of representation learning.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1162/neco_a_01234
