Convergence Results for Some Temporal Difference Methods Based on Least Squares
Name
Yu-2009-Convergence Results for Some Temporal Difference Methods Based on Least Squares.pdf
Size
1.32 MB
Format
Adobe PDF
Checksum (MD5)
414eb35aadec84756b7bcb9cf866811a
Author(s)
Yu, Huizhen
Bertsekas, Dimitri P.
Date Issued
July 2009
Journal
IEEE Transactions on Automatic Control
Publisher
Institute of Electrical and Electronics Engineers
Citation
Huizhen Yu, and D.P. Bertsekas. “Convergence Results for Some Temporal Difference Methods Based on Least Squares.” IEEE Transactions on Automatic Control 54.7 (2009): 1515–1531. Web.©2009 IEEE.
Version
Final published version
Abstract
We consider finite-state Markov decision processes, and prove convergence and rate of convergence results for certain least squares policy evaluation algorithms of the type known as LSPE(lambda ). These are temporal difference methods for constructing a linear function approximation of the cost function of a stationary policy, within the context of infinite-horizon discounted and average cost dynamic programming. We introduce an average cost method, patterned after the known discounted cost method, and we prove its convergence for a range of constant stepsize choices. We also show that the convergence rate of both the discounted and the average cost methods is optimal within the class of temporal difference methods. Analysis and experiment indicate that our methods are substantially and often dramatically faster than TD(lambda), as well as more reliable.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/tac.2009.2022097
