Composing Visual Relations with Composable Diffusion Models

Abstract

Humans are able to build complex representations of our world – representing the world as compositional combinations of both objects and their interdependent relations. Recent work in text-guided diffusion models have produced impressive results in generating photorealistic images, but such models often fail to capture spatial relationships between objects, and will often generate scenes where individual specified relations are incorrectly captured. An underlying cause is that such models are not explicitly compositional – when given a relational text description such as fork on plate or plate on fork, models will regress to generating the previously seen images, and will only generate images with a fork on a plate. We propose an approach to more accurately capture relations by decomposing the image probability density as a hierarchical product between lifted density representing abstract relations between objects and individual densities representing each object. We illustrate how this approach is simple to implement in practice and enables us to scale to accurately capture relations between objects across simulated and realistic scenes.