Visualization and Behavioral Testing Of Common Sense Generative Programs

Abstract

Probabilistic generative programs are powerful tools that allow for modeling complex 3D worlds containing objects and agents. Recent advances in these programs have resulted in creation of rich models whose traces represent 3D scenes, but there exist challenges in using visualizations and simulation tools for practical implementations. In this thesis, I describe the development of infrastructure to accelerate research in this area. Specifically, I present a pipeline for synthetic data generation with physics simulation capabilities and a suite of rendering options. By leveraging existing scene graph generators and multiple visualization engines, photorealistic datasets can be produced to evaluate probabilistic generative programs and create stimuli for gathering information on human behavior. This framework allows fine-grained temporal tracking of object poses and velocities, both with and without occlusion, facilitating the collection of rich human behavioral data on dynamic object tracking. More broadly, the tools developed here provide visualization, debugging capabilities, and configurable synthetic datasets to benchmark future progress in 3D scene understanding. Development of this infrastructure is an investment in improved synthetic data generation and analysis frameworks is an important step toward robust probabilistic generative programs for 3D world modeling.