Inverse reinforcement learning with locally consistent reward functions
Author(s)Nguyen, Quoc Phong; Low, Bryan Kian Hsiang; Jaillet, Patrick
MetadataShow full item record
Existing inverse reinforcement learning (IRL) algorithms have assumed each expert’s demonstrated trajectory to be produced by only a single reward function. This paper presents a novel generalization of the IRL problem that allows each trajectory to be generated by multiple locally consistent reward functions, hence catering to more realistic and complex experts’ behaviors. Solving our generalized IRL problem thus involves not only learning these reward functions but also the stochastic transitions between them at any state (including unvisited states). By representing our IRL problem with a probabilistic graphical model, an expectation-maximization (EM) algorithm can be devised to iteratively learn the different reward functions and the stochastic transitions between them in order to jointly improve the likelihood of the expert’s demonstrated trajectories. As a result, the most likely partition of a trajectory into segments that are generated from different locally consistent reward functions selected by EM can be derived. Empirical evaluation on synthetic and real-world datasets shows that our IRL algorithm outperforms the state-of-the-art EM clustering with maximum likelihood IRL, which is, interestingly, a reduced variant of our approach.
DepartmentMassachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Advances in Neural Information Processing Systems 28 (NIPS 2015)
Neural Information Processing Systems Foundation
Nguyen, Quoc Phong. Bryan Kian Hsiang Low, and Patrick Jaillet. "Inverse Reinforcement Learning with Locally Consistent Reward Functions." Advances in Neural Information Processing Systems 28 (NIPS 2015), 4-9 December 2015, Long Beach, CA, NIPS, 2015.
Final published version