Optimistic gittins indices
Name
6036-optimistic-gittins-indices.pdf
Size
1.77 MB
Format
Adobe PDF
Checksum (MD5)
45ae788b3d45b45a02f9819d73ca4169
Author(s)
Gutin, Eli
Farias, Vivek F.
Date Issued
December 2016
Journal
Advances in Neural Information Processing Systems 29 (NIPS 2016)
Publisher
NIPS Foundation
Citation
Gutin, Eli and Vivek F. Farias. "Optimistic gittins indices." Advances in Neural Information Processing Systems 29 (NIPS 2016), Barcelona, Spain, NIPS Foundation, December 2016. © 2016 NIPS Foundation
Version
Final published version
Abstract
Starting with the Thomspon sampling algorithm, recent years have seen a resurgence of interest in Bayesian algorithms for the Multi-armed Bandit (MAB) problem. These algorithms seek to exploit prior information on arm biases and while several have been shown to be regret optimal, their design has not emerged from a principled approach. In contrast, if one cared about Bayesian regret discounted over an infinite horizon at a fixed, pre-specified rate, the celebrated Gittins index theorem offers an optimal algorithm. Unfortunately, the Gittins analysis does not appear to carry over to minimizing Bayesian regret over all sufficiently large horizons and computing a Gittins index is onerous relative to essentially any incumbent index scheme for the Bayesian MAB problem. The present paper proposes a sequence of 'optimistic' approximations to the Gittins index. We show that the use of these approximations in concert with the use of an increasing discount factor appears to offer a compelling alternative to state-of-the-art index schemes proposed for the Bayesian MAB problem in recent years by offering substantially improved performance with little to no additional computational overhead. In addition, we prove that the simplest of these approximations yields frequentist regret that matches the Lai-Robbins lower bound, including achieving matching constants.
MIT Department
Sloan School of Management
Massachusetts Institute of Technology. Operations Research Center
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://papers.nips.cc/paper/6036-optimistic-gittins-indices
