Learning to Plan via Deep Optimistic Value Exploration
Author(s)Seyde, Tim; Schwarting, Wilko; Karaman, Sertac; Rus, Daniela L
MetadataShow full item record
Deep exploration requires coordinated long-term planning. We present a model-based reinforcement learning algorithm that guides policy learning through a value function that exhibits optimism in the face of uncertainty. We capture uncertainty over values by combining predictions from an ensemble of models and formulate an upper confidence bound (UCB) objective to recover optimistic estimates. Training the policy on ensemble rollouts with the learned value function as the terminal cost allows for projecting long-term interactions into a limited planning horizon, thus enabling deep optimistic exploration. We do not assume a priori knowledge of either the dynamics or reward function. We demonstrate that our approach can accommodate both dense and sparse reward signals, while improving sample complexity on a variety of benchmarking tasks. Keywords: Reinforcement Learning; Deep Exploration; Model-Based; Value Function; UCB
DepartmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Proceedings of Machine Learning Research
Seyde, Tim, et al. "Learning to Plan via Deep Optimistic Value Exploration." Proceedings of Machine Learning Research, 120 (August 2020), 1-14.
Author's final manuscript