Towards feature selection in actor-critic algorithms
Author(s)Rohanimanesh, Khashayar; Roy, Nicholas; Tedrake, Russell Louis
MetadataShow full item record
Choosing features for the critic in actor-critic algorithms with function approximation is known to be a challenge. Too few critic features can lead to degeneracy of the actor gradient, and too many features may lead to slower convergence of the learner. In this paper, we show that a wellstudied class of actor policies satisfy the known requirements for convergence when the actor features are selected carefully. We demonstrate that two popular representations for value methods - the barycentric interpolators and the graph Laplacian proto-value functions - can be used to represent the actor in order to satisfy these conditions. A consequence of this work is a generalization of the proto-value function methods to the continuous action actor-critic domain. Finally, we analyze the performance of this approach using a simulation of a torque-limited inverted pendulum.
URL to paper listed on conference page
DepartmentMassachusetts Institute of Technology. Department of Aeronautics and Astronautics; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Proceedings of Workshop on Abstraction in Reinforcement Learning, Joint workshop at ICML, UAI, and COLT 2009
Rohanimanesh, Khashayar, Nicholas Roy and Russ Tedrake. "Towards feature selection in actor-critic algorithms." in Proceedings of the ICML/UAI/COLT Workshop on Abstraction in Reinforcement Learning, Montreal, Canada, 2009.
Author's final manuscript