Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation
Name
6233-hierarchical-deep-reinforcement-learning-integrating-temporal-abstraction-and-intrinsic-motivation.pdf
Size
1.09 MB
Format
Adobe PDF
Checksum (MD5)
d64b0bd84316500ec9654b70ebeb4f59
Author(s)
Kulkarni, Tejas D.
Narasimhan, Karthik Rajagopal
Saeedi, Ardavan
Tenenbaum, Joshua B
Date Issued
December 2016
Journal
Advances in Neural Information Processing Systems 29 (NIPS 2016)
Publisher
Neural Information Processing Systems Foundation
Citation
Kulkarni, Tejas D. et al. "Hierarchical Deep Reinforcement Learning:
Integrating Temporal Abstraction and Intrinsic Motivation." Advances in Neural Information Processing Systems 29 (NIPS 2016), Barcelona, Spain, December 5-10, 2016. © 2016 Neural Information Processing Systems Foundation
Version
Final published version
Abstract
Learning goal-directed behavior in environments with sparse feedback is a major challenge for reinforcement learning algorithms. One of the key difficulties is insufficient exploration, resulting in an agent being unable to learn robust policies. Intrinsically motivated agents can explore new behavior for their own sake rather than to directly solve external goals. Such intrinsic behaviors could eventually help the agent solve tasks posed by the environment. We present hierarchical-DQN (h-DQN), a framework to integrate hierarchical action-value functions, operating at different temporal scales, with goal-driven intrinsically motivated deep reinforcement learning. A top-level q-value function learns a policy over intrinsic goals, while a lower-level function learns a policy over atomic actions to satisfy the given goals. h-DQN allows for flexible goal specifications, such as functions over entities and relations. This provides an efficient space for exploration in complicated environments. We demonstrate the strength of our approach on two problems with very sparse and delayed feedback: (1) a complex discrete stochastic decision process with stochastic transitions, and (2) the classic ATARI game - 'Montezuma's Revenge'.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://papers.nips.cc/paper/6233-hierarchical-deep-reinforcement-learning-integrating-temporal-abstraction-and-intrinsic-motivation
