Quickest change detection approach to optimal control in Markov decision processes with model changes
Name
1609.06757.pdf
Size
313.81 KB
Format
Adobe PDF
Checksum (MD5)
13cb0c73308aea0e1ea71d45cb48ecd8
Author(s)
Banerjee, Taposh
Liu, Miao
How, Jonathan P
Date Issued
July 2017
Journal
2017 American Control Conference (ACC)
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Banerjee, Taposh, Miao Liu, and Jonathan P. How. “Quickest Change Detection Approach to Optimal Control in Markov Decision Processes with Model Changes.” 2017 American Control Conference (ACC), May 2017, Seattle, WA, USA, 2017.
Version
Original manuscript
Abstract
Optimal control in non-stationary Markov decision processes (MDP) is a challenging problem. The aim in such a control problem is to maximize the long-term discounted reward when the transition dynamics or the reward function can change over time. When a prior knowledge of change statistics is available, the standard Bayesian approach to this problem is to reformulate it as a partially observable MDP (POMDP) and solve it using approximate POMDP solvers, which are typically computationally demanding. In this paper, the problem is analyzed through the viewpoint of quickest change detection (QCD), a set of tools for detecting a change in the distribution of a sequence of random variables. Current methods applying QCD to such problems only passively detect changes by following prescribed policies, without optimizing the choice of actions for long term performance. We demonstrate that ignoring the reward-detection trade-off can cause a significant loss in long term rewards, and propose a two threshold switching strategy to solve the issue. A non-Bayesian problem formulation is also proposed for scenarios where a Bayesian formulation cannot be defined. The performance of the proposed two threshold strategy is examined through numerical analysis on a non-stationary MDP task, and the strategy outperforms the state-of-the-art QCD methods in both Bayesian and non-Bayesian settings.
MIT Department
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.23919/ACC.2017.7962986
