Output-weighted sampling for multi-armed bandits with extreme payoffs
Name
yang-et-al-2022-output-weighted-sampling-for-multi-armed-bandits-with-extreme-payoffs.pdf
Size
1.67 MB
Format
Adobe PDF
Checksum (MD5)
05a7f916f064c893eb27e68334cca6bb
Author(s)
Yang, Yibo
Blanchard, Antoine
Sapsis, Themistoklis
Perdikaris, Paris
Date Issued
April 2022
Journal
Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences
Publisher
The Royal Society
Citation
Yang Yibo, Blanchard Antoine, Sapsis Themistoklis and Perdikaris Paris 2022Output-weighted sampling for multi-armed bandits with extreme payoffsProc. R. Soc. A.47820210781.
Version
Final published version
Abstract
We present a new type of acquisition function for online decision-making in multi-armed and contextual bandit problems with extreme payoffs. Specifically, we model the payoff function as a Gaussian process and formulate a novel type of upper confidence bound acquisition function that guides exploration towards the bandits that are deemed most relevant according to the variability of the observed rewards. This is achieved by computing a tractable likelihood ratio that quantifies the importance of the output relative to the inputs and essentially acts as anattention mechanismthat promotes exploration of extreme rewards. Our formulation is supported by asymptotic zero-regret guarantees, and its performance is demonstrated across several synthetic benchmarks, as well as two realistic examples involving noisy sensor network data. Finally, we provide a JAX library for efficient bandit optimization using Gaussian processes.
Subjects
General Physics and Astronomy
General Engineering
General Mathematics
MIT Department
Massachusetts Institute of Technology. Department of Mechanical Engineering
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1098/rspa.2021.0781
