Visual object networks: Image generation with disentangled 3D representation
Name
7297-visual-object-networks-image-generation-with-disentangled-3d-representations.pdf
Description
Published version
Size
4.71 MB
Format
Adobe PDF
Checksum (MD5)
ad17bb9b6eae6f213a585b998312856b
Author(s)
Zhu, Jun-Yan
Zhang, Zhoutong
Zhang, Chengkai
Wu, Jiajun
Torralba, Antonio
Tenenbaum, Joshua B.
Freeman, William T.
Date Issued
2018
Citation
Zhu, Jun-Yan, Zhang, Zhoutong, Zhang, Chengkai, Wu, Jiajun, Torralba, Antonio et al. 2018. "Visual object networks: Image generation with disentangled 3D representation."
Version
Final published version
Abstract
© 2018 Curran Associates Inc..All rights reserved. Recent progress in deep generative models has led to tremendous breakthroughs in image generation. However, while existing models can synthesize photorealistic images, they lack an understanding of our underlying 3D world. We present a new generative model, Visual Object Networks (VON), synthesizing natural images of objects with a disentangled 3D representation. Inspired by classic graphics rendering pipelines, we unravel our image formation process into three conditionally independent factors-shape, viewpoint, and texture-and present an end-to-end adversarial learning framework that jointly models 3D shapes and 2D images. Our model first learns to synthesize 3D shapes that are indistinguishable from real shapes. It then renders the object's 2.5D sketches (i.e., silhouette and depth map) from its shape under a sampled viewpoint. Finally, it learns to add realistic texture to these 2.5D sketches to generate natural images. The VON not only generates images that are more realistic than state-of-the-art 2D image synthesis methods, but also enables many 3D operations such as changing the viewpoint of a generated image, editing of shape and texture, linear interpolation in texture and shape space, and transferring appearance across different objects and viewpoints.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://papers.nips.cc/paper/7297-visual-object-networks-image-generation-with-disentangled-3d-representations
