Small nonlinearities in activation functions create bad local minima in neural networks
Author(s)
Yun, Chulee; Sra, Suvrit; Jadbabaie, Ali
DownloadSubmitted version (433.4Kb)
Open Access Policy
Open Access Policy
Creative Commons Attribution-Noncommercial-Share Alike
Terms of use
Metadata
Show full item recordAbstract
© 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved. We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with “slightest” nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general “no spurious local minima” is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima. We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum. Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks. We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.
Date issued
2019Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science; Massachusetts Institute of Technology. Department of Civil and Environmental Engineering; Massachusetts Institute of Technology. Institute for Data, Systems, and SocietyJournal
7th International Conference on Learning Representations, ICLR 2019
Citation
Yun, Chulee, Sra, Suvrit and Jadbabaie, Ali. 2019. "Small nonlinearities in activation functions create bad local minima in neural networks." 7th International Conference on Learning Representations, ICLR 2019.
Version: Original manuscript