Policy search approaches to reinforcement learning for quadruped locomotion

Abstract

Legged locomotion is a challenging problem for machine learning to solve. A quadruped has 12 degrees of freedom which results in a large state space for the resulting Markov Decision Problem (DP). It is too difficult for computers to completely learn the state space, while it is too difficult for humans to fully understand the system dynamics and directly program the most efficient controller. This thesis combines these two approaches by integrating a model-based controller approach with reinforcement learning to develop an effective walk for a quadruped robot. We then evaluate different policy search approaches to reinforcement learning. To solve the Partially Observable Markov Decision Problem (POMIDP), a deterministic simulation is developed that generates a model which allows us to conduct a direct. policy search using dynamic programming. This is compared against using a, nondeterministic simulation to generate a model that evaluates policies. We show that using deterministic transitions to allow the use of dynamic programming has little impact on the performance of our system. Two local policy search approaches are implemented.

(cont.) A hill climbing algorithm is compared to a policy gradient algorithm to optimize parameters for the robot's model-based controller. The optimal machine-learned policy achieved a 155'% increase in performance over the hand-tuned policy. The baseline hill climbing algorithm is shown to outperform the policy gradient. algorithm with this particular gait.