On the Convergence Rate of Incremental Aggregated Gradient Algorithms
Name
15m1049695.pdf
Size
354.95 KB
Format
Adobe PDF
Checksum (MD5)
9535424e05f18998cfaf3a4a101f2de8
Author(s)
Gurbuzbalaban, Mert
Koksal, Asuman E.
Parrilo, Pablo A
Date Issued
January 2017
Journal
SIAM Journal on Optimization
Publisher
Society for Industrial & Applied Mathematics (SIAM)
Citation
Gürbüzbalaban, M., et al. “On the Convergence Rate of Incremental Aggregated Gradient Algorithms.” SIAM Journal on Optimization, vol. 27, no. 2, Jan. 2017, pp. 1035–48. © 2017 Society for Industrial and Applied Mathematics.
Version
Final published version
Abstract
Motivated by applications to distributed optimization over networks and large-scale data processing in machine learning, we analyze the deterministic incremental aggregated gradient method for minimizing a finite sum of smooth functions where the sum is strongly convex. This method processes the functions one at a time in a deterministic order and incorporates a memory of previous gradient values to accelerate convergence. Empirically it performs well in practice; however, no theoretical analysis with explicit rate results was previously given in the literature to our knowledge, in particular most of the recent efforts concentrated on the randomized versions. In this paper, we show that this deterministic algorithm has global linear convergence and we characterize the convergence rate. We also consider an aggregated method with momentum and demonstrate its linear convergence. Our proofs rely on a careful choice of a Lyapunov function that offers insight into the algorithm's behavior and simplifies the proofs considerably. Key words: convex optimization, first-order methods, convergence analysis, large-scale optimization
MIT Department
Massachusetts Institute of Technology. Department of Biological Engineering
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1137/15M1049695
