On the limitations of first-order approximation in GAN dynamics
Name
1706.09884.pdf
Description
Submitted version
Size
786.21 KB
Format
Adobe PDF
Checksum (MD5)
efb7a86c701da67a7c13a138a4b36c11
Author(s)
Madry, Aleksander
Date Issued
2018
Citation
Madry, Aleksander. 2018. "On the limitations of first-order approximation in GAN dynamics."
Version
Original manuscript
Abstract
© Copyright 2018 by the author(s). While Generative Adversarial Networks (GANs) have demonstrated promising performance on multiple vision tasks, their learning dynamics are not yet well understood, both in theory and in practice. To address this issue, we study GAN dynamics in a simple yet rich parametric model that exhibits several of the common problematic convergence behaviors such as vanishing gradients, mode collapse, and diverging or oscillatory behavior. In spite of the non-convex nature of our model, we are able to perform a rigorous theoretical analysis of its convergence behavior. Our analysis reveals an interesting dichotomy, a GAN with an optimal discriminator provably converges, while first order approximations of the discriminator steps lead to unstable GAN dynamics and mode collapse. Our result suggests that using first order discriminator steps (the de-facto standard in most existing GAN setups) might be one of the factors that makes GAN training challenging in practice.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://openreview.net/forum?id=HJYQLb-RW
