Direct Runge-Kutta discretization achieves acceleration
Name
NeurIPS-2018-direct-runge-kutta-discretization-achieves-acceleration-Paper.pdf
Description
Published version
Size
593.16 KB
Format
Unknown
Checksum (MD5)
0013d1fd8c33e8a7ab9dd17584200a75
Author(s)
Zhang, Jingzhao
Mokhtari, Aryan
Sra, Suvrit
Jadbabaie-Moghadam, Ali
Date Issued
2018
Journal
Advances in Neural Information Processing Systems
Citation
Zhang, Jingzhao, Mokhtari, Aryan, Sra, Suvrit and Jadbabaie, Ali. 2018. "Direct Runge-Kutta discretization achieves acceleration." Advances in Neural Information Processing Systems, 2018-December.
Version
Final published version
Abstract
© 2018 Curran Associates Inc..All rights reserved. We study gradient-based optimization methods obtained by directly discretizing a second-order ordinary differential equation (ODE) related to the continuous limit of Nesterov's accelerated gradient method. When the function is smooth enough, we show that acceleration can be achieved by a stable discretization of this ODE using standard Runge-Kutta integrators. Specifically, we prove that under Lipschitz-gradient, convexity and order-(s + 2) differentiability assumptions, the sequence of iterates generated by discretizing the proposed second-order ODE converges to the optimal solution at a rate of O(N−2 s+1 s ), where s is the order of the Runge-Kutta numerical integrator. Furthermore, we introduce a new local flatness condition on the objective, under which rates even faster than O(N−2) can be achieved with low-order integrators and only gradient information. Notably, this flatness condition is satisfied by several standard loss functions used in machine learning. We provide numerical experiments that verify the theoretical rates predicted by our results.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
