Efficiently testing local optimality and escaping saddles for RELu networks
Name
1809.10858.pdf
Description
Accepted version
Size
442.52 KB
Format
Adobe PDF
Checksum (MD5)
dd1c5973d9e9f0be39467847fc26f759
Author(s)
Jadbabaie, Ali
Yun, Chulhee
Sra, Suvrit
Date Issued
2019
Journal
7th International Conference on Learning Representations, ICLR 2019
Citation
Jadbabaie, Ali, Yun, Chulhee and Sra, Suvrit. 2019. "Efficiently testing local optimality and escaping saddles for RELu networks." 7th International Conference on Learning Representations, ICLR 2019.
Version
Author's final manuscript
Abstract
© 7th International Conference on Learning Representations, ICLR 2019. All Rights Reserved. We provide a theoretical algorithm for checking local optimality and escaping saddles at nondifferentiable points of empirical risks of two-layer ReLU networks. Our algorithm receives any parameter value and returns: local minimum, second-order stationary point, or a strict descent direction. The presence of M data points on the nondifferentiability of the ReLU divides the parameter space into at most 2M regions, which makes analysis difficult. By exploiting polyhedral geometry, we reduce the total computation down to one convex quadratic program (QP) for each hidden node, O(M) (in)equality tests, and one (or a few) nonconvex QP. For the last QP, we show that our specific problem can be solved efficiently, in spite of nonconvexity. In the benign case, we solve one equality constrained QP, and we prove that projected gradient descent solves it exponentially fast. In the bad case, we have to solve a few more inequality constrained QPs, but we prove that the time complexity is exponential only in the number of inequality constraints. Our experiments show that either benign case or bad case with very few inequality constraints occurs, implying that our algorithm is efficient in most cases.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://openreview.net/forum?id=HylTXn0qYX
