Learning to see physics via visual de-animation
Author(s)
Wu, Jiajun; Lu, Erika; Kohli, Pushmeet; Freeman, William T; Tenenbaum, Joshua B
DownloadPublished version (1.111Mb)
Terms of use
Metadata
Show full item recordAbstract
We introduce a paradigm for understanding physical scenes without human annotations. At the core of our system is a physical world representation that is first recovered by a perception module and then utilized by physics and graphics engines. During training, the perception module and the generative models learn by visual de-animation - interpreting and reconstructing the visual information stream. During testing, the system first recovers the physical world state, and then uses the generative models for reasoning and future prediction. Even more so than forward simulation, inverting a physics or graphics engine is a computationally hard problem; we overcome this challenge by using a convolutional inversion network. Our system quickly recognizes the physical world state from appearance and motion cues, and has the flexibility to incorporate both differentiable and non-differentiable physics and graphics engines. We evaluate our system on both synthetic and real datasets involving multiple physical scenes, and demonstrate that our system performs well on both physical state estimation and reasoning problems. We further show that the knowledge learned on the synthetic dataset generalizes to constrained real images.
Date issued
2017-12Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence LaboratoryJournal
Advances in Neural Information Processing Systems 30 (NIPS 2017)
Publisher
Neural Information Processing Systems Foundation, Inc
Citation
Wu, Jiajun et al. "Learning to see physics via visual de-animation." Advances in Neural Information Processing Systems 30 (NIPS 2017), December 2017, Long Beach, California, Neural Information Processing Systems Foundation, Inc, December 2017. © 2017 Neural Information Processing Systems Foundation, Inc.
Version: Final published version