Direct optimization through arg max for discrete variational auto-encoder
Author(s)
Gane, Andreea; Jaakkola, Tommi S
DownloadPublished version (2.451Mb)
Publisher Policy
Publisher Policy
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Terms of use
Metadata
Show full item recordAbstract
Reparameterization of variational auto-encoders with continuous random variables is an effective method for reducing the variance of their gradient estimates. In the discrete case, one can perform reparametrization using the Gumbel-Max trick, but the resulting objective relies on an arg max operation and is non-differentiable. In contrast to previous works which resort to softmax-based relaxations, we propose to optimize it directly by applying the direct loss minimization approach. Our proposal extends naturally to structured discrete latent variable models when evaluating the arg max operation is tractable. We demonstrate empirically the effectiveness of the direct loss minimization technique in variational autoencoders with both unstructured and structured discrete latent variables.
Date issued
2019-12Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer ScienceJournal
33rd Conference on Neural Information Processing Systems (NeurIPS 2019)
Publisher
Morgan Kaufmann Publishers
Citation
Lorberbom, Guy et al. “Direct optimization through arg max for discrete variational auto-encoder.” 33rd Conference on Neural Information Processing Systems, December 2019, Vancouver, Canada, Morgan Kaufmann Publishers, 2019. © 2019 The Author(s)
Version: Final published version
ISSN
1049-5258