Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism
Name
cheung20a.pdf
Description
Published version
Size
921.42 KB
Format
Adobe PDF
Checksum (MD5)
5b56d5d51611689ae8f190fcf8f10e98
Author(s)
Cheung, Wang Chi
Simchi-Levi, David
Zhu, Ruihao
Date Issued
2020
Journal
INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 119
Citation
Cheung, Wang Chi, Simchi-Levi, David and Zhu, Ruihao. 2020. "Reinforcement Learning for Non-Stationary Markov Decision Processes: The Blessing of (More) Optimism." INTERNATIONAL CONFERENCE ON MACHINE LEARNING, VOL 119, 119.
Version
Final published version
Abstract
We consider un-discounted reinforcement learning (RL) in Markov decision processes (MDPs)
under drifting non-stationarity, i.e., both the reward and state transition distributions are allowed
to evolve over time, as long as their respective
total variations, quantified by suitable metrics, do
not exceed certain variation budgets. We first
develop the Sliding Window Upper-Confidence
bound for Reinforcement Learning with Confidence Widening (SWUCRL2-CW) algorithm, and
establish its dynamic regret bound when the variation budgets are known. In addition, we propose
the Bandit-over-Reinforcement Learning (BORL)
algorithm to adaptively tune the SWUCRL2-CW
algorithm to achieve the same dynamic regret
bound, but in a parameter-free manner, i.e., without knowing the variation budgets. Notably, learning non-stationary MDPs via the conventional optimistic exploration technique presents a unique
challenge absent in existing (non-stationary) bandit learning settings. We overcome the challenge
by a novel confidence widening technique that
incorporates additional optimism.
MIT Department
Massachusetts Institute of Technology. Institute for Data, Systems, and Society
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://proceedings.mlr.press/v119/cheung20a.html
