Galileo: Perceiving physical object properties by integrating a physics engine with deep learning
Author(s)
Yildirim, Ilker
Wu, Jiajun
Lim, Joseph Jaewhan
Tenenbaum, Joshua B.
Freeman, William T.
Date Issued
December 2015
Journal
Advances in Neural Information Processing Systems 28 (NIPS 2015)
Publisher
Neural Information Processing Systems Foundation
Citation
Wu, Jianjun et al. "Galileo: Perceiving Physical Object Properties by Integrating a Physics Engine with Deep Learning." Advances in Neural Information Processing Systems 28 (NIPS 2015), 7-12 December, 2015, Montreal, Canada, Neural Information Processing Systems Foundation, 2015. © 2015 Neural Information Processing Systems Foundation
Version
Final published version
Abstract
Humans demonstrate remarkable abilities to predict physical events in dynamic scenes, and to infer the physical properties of objects from static images. We propose a generative model for solving these problems of physical scene understanding from real-world videos and images. At the core of our generative model is a 3D physics engine, operating on an object-based representation of physical properties, including mass, position, 3D shape, and friction. We can infer these latent properties using relatively brief runs of MCMC, which drive simulations in the physics engine to fit key features of visual observations. We further explore directly mapping visual inputs to physical properties, inverting a part of the generative process using deep learning. We name our model Galileo, and evaluate it on a video dataset with simple yet physically rich scenarios. Results show that Galileo is able to infer the physical properties of objects and predict the outcome of a variety of physical events, with an accuracy comparable to human subjects. Our study points towards an account of human vision with generative physical knowledge at its core, and various recognition models as helpers leading to efficient inference.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://papers.nips.cc/paper/5780-galileo-perceiving-physical-object-properties-by-integrating-a-physics-engine-with-deep-learning
