Robustness and Adaptation via a Generative Model of Policies in Reinforcement Learning
Name
Derek-kderek-meng-eecs-2021-thesis.pdf
Description
Thesis PDF
Size
14.77 MB
Format
Adobe PDF
Checksum (MD5)
4ee715e54c7bfe2233433d9e9f2d8f5b
Author(s)
Derek, Kenneth
Advisor(s)
Isola, Phillip
Date Issued
June 2021
Publisher
Massachusetts Institute of Technology
Abstract
In the natural world, life has found an uncountable number of ways to survive and often thrive. Between and even within species, each individual has a slightly unique way of existing, and this diversity lends robustness to life in general. In this work, we aim to incentivize diversity of agent policies while optimizing for an external reward. To this end, we introduce a generative model of policies which maps a low-dimensional latent space to an agent policy space. In order to learn a broad range of solutions, our generative model uses a diversity regularizer that incentivizes different agent behaviors given the same state. Agents are assigned a specific latent vector persistent throughout their trajectory, and the generator learns to encode behavioral preferences in the latent space. Results show that our generator is able to find an array of policies that can express agent individuality through distinct and unique agent policies. Of particular interest, we find that having a diverse policy space allows us to rapidly adapt to unforeseen environmental ablations simply by optimizing generated policies in the low-dimensional latent space. We test this adaptability in an open-ended grid-world, as well as in a competitive, zero-sum, two-player soccer environment.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright MIT
Persistent DSpace Link
