## Bayesian Nonparametric Inverse Reinforcement Learning

##### Author(s)

How, Jonathan P.; Michini, Bernard J.
DownloadHow_Bayesian nonparametric.pdf (427.1Kb)

OPEN_ACCESS_POLICY

# Open Access Policy

Creative Commons Attribution-Noncommercial-Share Alike

##### Terms of use

##### Metadata

Show full item record##### Abstract

Inverse reinforcement learning (IRL) is the task of learning the reward function of a Markov Decision Process (MDP) given the transition function and a set of observed demonstrations in the form of state-action pairs. Current IRL algorithms attempt to find a single reward function which explains the entire observation set. In practice, this leads to a computationally-costly search over a large (typically infinite) space of complex reward functions. This paper proposes the notion that if the observations can be partitioned into smaller groups, a class of much simpler reward functions can be used to explain each group. The proposed method uses a Bayesian nonparametric mixture model to automatically partition the data and find a set of simple reward functions corresponding to each partition. The simple rewards are interpreted intuitively as subgoals, which can be used to predict actions or analyze which states are important to the demonstrator. Experimental results are given for simple examples showing comparable performance to other IRL algorithms in nominal situations. Moreover, the proposed method handles cyclic tasks (where the agent begins and ends in the same state) that would break existing algorithms without modification. Finally, the new algorithm has a fundamentally different structure than previous methods, making it more computationally efficient in a real-world learning scenario where the state space is large but the demonstration set is small.

##### Date issued

2012-09##### Department

Massachusetts Institute of Technology. Department of Aeronautics and Astronautics##### Journal

Machine Learning and Knowledge Discovery in Databases

##### Publisher

Springer-Verlag

##### Citation

Michini, Bernard, and Jonathan P. How. Bayesian Nonparametric Inverse Reinforcement Learning. Springer-Verlag, 2012.

Version: Author's final manuscript

##### ISBN

978-3-642-33485-6

978-3-642-33486-3

##### ISSN

0302-9743

1611-3349