A Structured Multiarmed Bandit Problem and the Greedy Policy
Author(s)Rusmevichientong, Paat; Mersereau, Adam J.; Tsitsiklis, John N.
MetadataShow full item record
We consider a multiarmed bandit problem where the expected reward of each arm is a linear function of an unknown scalar with a prior distribution. The objective is to choose a sequence of arms that maximizes the expected total (or discounted total) reward. We demonstrate the effectiveness of a greedy policy that takes advantage of the known statistical correlation structure among the arms. In the infinite horizon discounted reward setting, we show that the greedy and optimal policies eventually coincide, and both settle on the best arm. This is in contrast with the Incomplete Learning Theorem for the case of independent arms. In the total reward setting, we show that the cumulative Bayes risk after T periods under the greedy policy is at most O(logT), which is smaller than the lower bound of Omega(log[superscript 2] T) established by Lai for a general, but different, class of bandit problems. We also establish the tightness of our bounds. Theoretical and numerical results show that the performance of our policy scales independently of the number of arms.
DepartmentMassachusetts Institute of Technology. Department of Electrical Engineering and Computer Science; Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
IEEE Transactions on Automatic Control
Institute of Electrical and Electronics Engineers
Mersereau, A.J., P. Rusmevichientong, and J.N. Tsitsiklis. “A Structured Multiarmed Bandit Problem and the Greedy Policy.” Automatic Control, IEEE Transactions on 54.12 (2009): 2787-2802. © 2009 Institute of Electrical and Electronics Engineers.
Final published version
Markov decision process (MDP)