Small nonlinearities in activation functions create bad local minima in neural networks
Name
1802.03487.pdf
Description
Submitted version
Size
433.45 KB
Format
Adobe PDF
Checksum (MD5)
6a06c8d01d82087d3ef3c5b731178aa9
Author(s)
Yun, Chulee
Sra, Suvrit
Jadbabaie, Ali
Date Issued
2019
Journal
7th International Conference on Learning Representations, ICLR 2019
Citation
Yun, Chulee, Sra, Suvrit and Jadbabaie, Ali. 2019. "Small nonlinearities in activation functions create bad local minima in neural networks." 7th International Conference on Learning Representations, ICLR 2019.
Version
Original manuscript
Abstract
© 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved. We investigate the loss surface of neural networks. We prove that even for one-hidden-layer networks with “slightest” nonlinearity, the empirical risks have spurious local minima in most cases. Our results thus indicate that in general “no spurious local minima” is a property limited to deep linear networks, and insights obtained from linear networks may not be robust. Specifically, for ReLU(-like) networks we constructively prove that for almost all practical datasets there exist infinitely many local minima. We also present a counterexample for more general activations (sigmoid, tanh, arctan, ReLU, etc.), for which there exists a bad local minimum. Our results make the least restrictive assumptions relative to existing results on spurious local optima in neural networks. We complete our discussion by presenting a comprehensive characterization of global optimality for deep linear networks, which unifies other results on this topic.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Department of Civil and Environmental Engineering
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
