A Structured Multiarmed Bandit Problem and the Greedy Policy
Name
Mersereau-2009-A Structured Multiar.pdf
Size
839.85 KB
Format
Adobe PDF
Checksum (MD5)
55927548263efdfa88ad57157e7b4c3e
Author(s)
Rusmevichientong, Paat
Mersereau, Adam J.
Tsitsiklis, John N.
Date Issued
December 2009
Journal
IEEE Transactions on Automatic Control
Publisher
Institute of Electrical and Electronics Engineers
Citation
Mersereau, A.J., P. Rusmevichientong, and J.N. Tsitsiklis. “A Structured Multiarmed Bandit Problem and the Greedy Policy.” Automatic Control, IEEE Transactions on 54.12 (2009): 2787-2802. © 2009 Institute of Electrical and Electronics Engineers.
Version
Final published version
Abstract
We consider a multiarmed bandit problem where the expected reward of each arm is a linear function of an unknown scalar with a prior distribution. The objective is to choose a sequence of arms that maximizes the expected total (or discounted total) reward. We demonstrate the effectiveness of a greedy policy that takes advantage of the known statistical correlation structure among the arms. In the infinite horizon discounted reward setting, we show that the greedy and optimal policies eventually coincide, and both settle on the best arm. This is in contrast with the Incomplete Learning Theorem for the case of independent arms. In the total reward setting, we show that the cumulative Bayes risk after T periods under the greedy policy is at most O(logT), which is smaller than the lower bound of Omega(log[superscript 2] T) established by Lai for a general, but different, class of bandit problems. We also establish the tightness of our bounds. Theoretical and numerical results show that the performance of our policy scales independently of the number of arms.
Subjects
Markov decision process (MDP)
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
http://dx.doi.org/10.1109/tac.2009.2031725
