Proximal algorithms and temporal difference methods for solving fixed point problems
Name
10589_2018_9990_ReferencePDF.pdf
Size
2.91 MB
Format
Adobe PDF
Checksum (MD5)
4552905f9db0ab0614620dd9a335a301
Author(s)
Bertsekas, Dimitri P
Date Issued
March 2, 2018
Publisher
Springer US
Version
Author's final manuscript
Abstract
Abstract
In this paper we consider large fixed point problems and solution with proximal algorithms. We show that for linear problems there is a close connection between proximal iterations, which are prominent in numerical analysis and optimization, and multistep methods of the temporal difference type such as TD(
$$\lambda $$
λ
), LSTD(
$$\lambda $$
λ
), and LSPE(
$$\lambda $$
λ
), which are central in simulation-based exact and approximate dynamic programming. One benefit of this connection is a new and simple way to accelerate the standard proximal algorithm by extrapolation towards a multistep iteration, which generically has a faster convergence rate. Another benefit is the potential for integration into the proximal algorithmic context of several new ideas that have emerged in the approximate dynamic programming context, including simulation-based implementations. Conversely, the analytical and algorithmic insights from proximal algorithms can be brought to bear on the analysis and the enhancement of temporal difference methods. We also generalize our linear case result to nonlinear problems that involve a contractive mapping, thus providing guaranteed and potentially substantial acceleration of the proximal and forward backward splitting algorithms at no extra cost. Moreover, under certain monotonicity assumptions, we extend the connection with temporal difference methods to nonlinear problems through a linearization approach.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/s10589-018-9990-5
