Reinforcement learning with limited reinforcement: Using Bayes risk for active learning in POMDPs
Name
fdoshi-aij12 (1).pdf
Size
297.61 KB
Format
Adobe PDF
Checksum (MD5)
78ce189f0668eb74b006b625b605f578
Author(s)
Pineau, Joelle
Doshi-Velez, Finale P
Roy, Nicholas
Date Issued
April 2012
Journal
Artificial Intelligence
Publisher
Elsevier
Citation
Doshi-Velez, Finale; Pineau, Joelle and Roy, Nicholas. “Reinforcement Learning with Limited Reinforcement: Using Bayes Risk for Active Learning in POMDPs.” Artificial Intelligence 187–188 (August 2012): 115–132. © 2012 Elsevier B.V.
Version
Author's final manuscript
Abstract
Acting in domains where an agent must plan several steps ahead to achieve a goal can be a challenging task, especially if the agentʼs sensors provide only noisy or partial information. In this setting, Partially Observable Markov Decision Processes (POMDPs) provide a planning framework that optimally trades between actions that contribute to the agentʼs knowledge and actions that increase the agentʼs immediate reward. However, the task of specifying the POMDPʼs parameters is often onerous. In particular, setting the immediate rewards to achieve a desired balance between information-gathering and acting is often not intuitive.
In this work, we propose an approximation based on minimizing the immediate Bayes risk for choosing actions when transition, observation, and reward models are uncertain. The Bayes-risk criterion avoids the computational intractability of solving a POMDP with a multi-dimensional continuous state space; we show it performs well in a variety of problems. We use policy queries—in which we ask an expert for the correct action—to infer the consequences of a potential pitfall without experiencing its effects. More important for human–robot interaction settings, policy queries allow the agent to learn the reward model without the reward values ever being specified.
MIT Department
Massachusetts Institute of Technology. Department of Aeronautics and Astronautics
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-NonCommercial-NoDerivs License
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1016/j.artint.2012.04.006
