Learning to Simulate Dynamic Environments With GameGAN
Name
2005.12126.pdf
Description
Published version
Size
4.8 MB
Format
Adobe PDF
Checksum (MD5)
2ca4a767bd41b3d1ecadd3f48d1bc0ba
Author(s)
Kim, Seung Wook
Zhou, Yuhao
Philion, Jonah
Torralba, Antonio
Fidler, Sanja
Date Issued
2020
Journal
Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Kim, Seung Wook, Zhou, Yuhao, Philion, Jonah, Torralba, Antonio and Fidler, Sanja. 2020. "Learning to Simulate Dynamic Environments With GameGAN." Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition.
Version
Original manuscript
Abstract
© 2020 IEEE. Simulation is a crucial component of any robotic system. In order to simulate correctly, we need to write complex rules of the environment: how dynamic agents behave, and how the actions of each of the agents affect the behavior of others. In this paper, we aim to learn a simulator by simply watching an agent interact with an environment. We focus on graphics games as a proxy of the real environment. We introduce GameGAN, a generative model that learns to visually imitate a desired game by ingesting screenplay and keyboard actions during training. Given a key pressed by the agent, GameGAN 'renders' the next screen using a carefully designed generative adversarial network. Our approach offers key advantages over existing work: we design a memory module that builds an internal map of the environment, allowing for the agent to return to previously visited locations with high visual consistency. In addition, GameGAN is able to disentangle static and dynamic components within an image making the behavior of the model more interpretable, and relevant for downstream tasks that require explicit reasoning over dynamic elements. This enables many interesting applications such as swapping different components of the game to build new games that do not exist. We will release the code and trained model, enabling human players to play generated games and their variations with our GameGAN.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/CVPR42600.2020.00131
