Bayesian scene understanding with object-based latent representation and multi-modal sensor fusion

Abstract

Scene understanding systems transform observations of an environment into a representation that facilitates reasoning over that environment. In this context, many reasoning tasks benefit from a high-level, object-based scene representation; quantification of uncertainty; and multi-modal sensor fusion. Here, we present a method for scene understanding that achieves all three of these desiderata. First, we introduce a generative probabilistic model that couples an object-based latent scene representation with multiple observations of different modalities. We then provide an inference procedure that draws samples from the posterior distribution over scene representations given observations and their associated camera parameters. Finally, we demonstrate that this method recovers accurate, object-based representations of scenes, and provides uncertainty quantification at a high level of abstraction.