Reinforcement learning with misspecified model classes
Name
How_Reinforcement learning.pdf
Size
728.46 KB
Format
Adobe PDF
Checksum (MD5)
e5262d6bf9bee3898013f447b175b934
Author(s)
Joseph, Joshua Mason
Geramifard, Alborz
Roberts, John W.
How, Jonathan P.
Roy, Nicholas
Date Issued
May 2013
Journal
Proceedings of the 2013 IEEE International Conference on Robotics and Automation
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Joseph, Joshua, Alborz Geramifard, John W. Roberts, Jonathan P. How, and Nicholas Roy. “Reinforcement Learning with Misspecified Model Classes.” 2013 IEEE International Conference on Robotics and Automation (May 2013).
Version
Author's final manuscript
Abstract
Real-world robots commonly have to act in complex, poorly understood environments where the true world dynamics are unknown. To compensate for the unknown world dynamics, we often provide a class of models to a learner so it may select a model, typically using a minimum prediction error metric over a set of training data. Often in real-world domains the model class is unable to capture the true dynamics, due to either limited domain knowledge or a desire to use a small model. In these cases we call the model class misspecified, and an unfortunate consequence of misspecification is that even with unlimited data and computation there is no guarantee the model with minimum prediction error leads to the best performing policy. In this work, our approach improves upon the standard maximum likelihood model selection metric by explicitly selecting the model which achieves the highest expected reward, rather than the most likely model. We present an algorithm for which the highest performing model from the model class is guaranteed to be found given unlimited data and computation. Empirically, we demonstrate that our algorithm is often superior to the maximum likelihood learner in a batch learning setting for two common RL benchmark problems and a third real-world system, the hydrodynamic cart-pole, a domain whose complex dynamics cannot be known exactly.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Aeronautics and Astronautics
Massachusetts Institute of Technology. Laboratory for Information and Decision Systems
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/ICRA.2013.6630686
