Universal Reinforcement Learning
Name
Farias-2009-Universal Reinforcement Learning.pdf
Size
338.39 KB
Format
Adobe PDF
Checksum (MD5)
466e61452a160e731bbb3c1f5a5a9590
Author(s)
Farias, Vivek F.
Moallemi, Ciamac C.
Van Roy, Benjamin
Weissman, Tsachy
Date Issued
April 2010
Journal
IEEE Transactions on Information Theory
Publisher
Institute of Electrical and Electronics Engineers
Citation
Farias, V.F. et al. “Universal Reinforcement Learning.” Information Theory, IEEE Transactions on 56.5 (2010): 2441-2454. © Copyright 2010 IEEE
Version
Final published version
Abstract
We consider an agent interacting with an unmodeled environment. At each time, the agent makes an observation, takes an action, and incurs a cost. Its actions can influence future observations and costs. The goal is to minimize the long-term average cost. We propose a novel algorithm, known as the active LZ algorithm, for optimal control based on ideas from the Lempel-Ziv scheme for universal data compression and prediction. We establish that, under the active LZ algorithm, if there exists an integer K such that the future is conditionally independent of the past given a window of K consecutive actions and observations, then the average cost converges to the optimum. Experimental results involving the game of Rock-Paper-Scissors illustrate merits of the algorithm.
Subjects
value iteration
reinforcement learning
optimal control
dynamic programming
Lempel-Ziv
Context tree
MIT Department
Sloan School of Management
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/TIT.2010.2043762
