Reasoning about objects under full occlusion

Abstract

While state-of-the-art machine learning models can outperform humans on certain tasks, most of them generalize poorly across domains and cannot reason about complex scenes. In this paper, we attempt to resolve this shortcoming by incorporating a physics engine as a prior for scene understanding. We test our approach on two computer vision tasks -- pose estimation and object matching -- under full occlusion, and demonstrate superior performance over state-of-the-art methods. We also present a preliminary case study which demonstrates that our model is consistent with human behavior. Our work demonstrates a successful approach to a novel and challenging task, provides a general framework to infer latent factors of scene via physics simulation, and extends support for intuitive physics-based approaches for robust visual reasoning.