Bayesian Nonparametric Inverse Reinforcement Learning
Name
How_Bayesian nonparametric.pdf
Size
427.11 KB
Format
Adobe PDF
Checksum (MD5)
e1a16553c148931687cee72e99452c01
Author(s)
How, Jonathan P.
Michini, Bernard J.
Date Issued
September 2012
Journal
Machine Learning and Knowledge Discovery in Databases
Publisher
Springer-Verlag
Citation
Michini, Bernard, and Jonathan P. How. Bayesian Nonparametric Inverse Reinforcement Learning. Springer-Verlag, 2012.
Version
Author's final manuscript
Abstract
Inverse reinforcement learning (IRL) is the task of learning the reward function of a Markov Decision Process (MDP) given the transition function and a set of observed demonstrations in the form of state-action pairs. Current IRL algorithms attempt to find a single reward function which explains the entire observation set. In practice, this leads to a computationally-costly search over a large (typically infinite) space of complex reward functions. This paper proposes the notion that if the observations can be partitioned into smaller groups, a class of much simpler reward functions can be used to explain each group. The proposed method uses a Bayesian nonparametric mixture model to automatically partition the data and find a set of simple reward functions corresponding to each partition. The simple rewards are interpreted intuitively as subgoals, which can be used to predict actions or analyze which states are important to the demonstrator. Experimental results are given for simple examples showing comparable performance to other IRL algorithms in nominal situations. Moreover, the proposed method handles cyclic tasks (where the agent begins and ends in the same state) that would break existing algorithms without modification. Finally, the new algorithm has a fundamentally different structure than previous methods, making it more computationally efficient in a real-world learning scenario where the state space is large but the demonstration set is small.
MIT Department
Massachusetts Institute of Technology. Department of Aeronautics and Astronautics
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike 3.0
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/978-3-642-33486-3_10
