Learning to Plan via Deep Optimistic Value Exploration
Name
L4DC2020_Final_Appendix.pdf
Size
1.08 MB
Format
Adobe PDF
Checksum (MD5)
54d112a924da6f06fce2253228b0b485
Author(s)
Seyde, Tim
Schwarting, Wilko
Karaman, Sertac
Rus, Daniela L
Date Issued
August 2020
Journal
Proceedings of Machine Learning Research
Citation
Seyde, Tim, et al. "Learning to Plan via Deep Optimistic Value Exploration." Proceedings of Machine Learning Research, 120 (August 2020), 1-14.
Version
Author's final manuscript
Abstract
Deep exploration requires coordinated long-term planning. We present a model-based reinforcement learning algorithm that guides policy learning through a value function that exhibits optimism in the face of uncertainty. We capture uncertainty over values by combining predictions from an ensemble of models and formulate an upper confidence bound (UCB) objective to recover optimistic estimates. Training the policy on ensemble rollouts with the learned value function as the terminal cost allows for projecting long-term interactions into a limited planning horizon, thus enabling deep optimistic exploration. We do not assume a priori knowledge of either the dynamics or reward function. We demonstrate that our approach can accommodate both dense and sparse reward signals, while improving sample complexity on a variety of benchmarking tasks. Keywords: Reinforcement Learning; Deep Exploration; Model-Based; Value Function; UCB
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
