Reinforcement learning with misspecified model classes
Author(s)
Joseph, Joshua Mason; Geramifard, Alborz; Roberts, John W.; How, Jonathan P.; Roy, Nicholas
DownloadHow_Reinforcement learning.pdf (728.4Kb)
OPEN_ACCESS_POLICY
Open Access Policy
Creative Commons Attribution-Noncommercial-Share Alike
Terms of use
Metadata
Show full item recordAbstract
Real-world robots commonly have to act in complex, poorly understood environments where the true world dynamics are unknown. To compensate for the unknown world dynamics, we often provide a class of models to a learner so it may select a model, typically using a minimum prediction error metric over a set of training data. Often in real-world domains the model class is unable to capture the true dynamics, due to either limited domain knowledge or a desire to use a small model. In these cases we call the model class misspecified, and an unfortunate consequence of misspecification is that even with unlimited data and computation there is no guarantee the model with minimum prediction error leads to the best performing policy. In this work, our approach improves upon the standard maximum likelihood model selection metric by explicitly selecting the model which achieves the highest expected reward, rather than the most likely model. We present an algorithm for which the highest performing model from the model class is guaranteed to be found given unlimited data and computation. Empirically, we demonstrate that our algorithm is often superior to the maximum likelihood learner in a batch learning setting for two common RL benchmark problems and a third real-world system, the hydrodynamic cart-pole, a domain whose complex dynamics cannot be known exactly.
Date issued
2013-05Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Aeronautics and Astronautics; Massachusetts Institute of Technology. Laboratory for Information and Decision SystemsJournal
Proceedings of the 2013 IEEE International Conference on Robotics and Automation
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Joseph, Joshua, Alborz Geramifard, John W. Roberts, Jonathan P. How, and Nicholas Roy. “Reinforcement Learning with Misspecified Model Classes.” 2013 IEEE International Conference on Robotics and Automation (May 2013).
Version: Author's final manuscript
ISBN
978-1-4673-5643-5
978-1-4673-5641-1
ISSN
1050-4729