Inverse Inverse Graphics

Abstract

To be human is to express—not only facts about the world, but also experiences, emotions, and ideas. We paint, explain, compose, persuade, dance, teach, sculpt, and sing, not merely to communicate bits of information from person to person, but rather to fan the windmills of one another's minds.

Well then, how do we do it? This thesis suggests that when engaging in artistic expression, we reflect on our shared perceptual and cognitive faculties—"common sense," so to speak—and then construct stimuli to evoke experiences in our audience's brains. Drawing on ideas from computer graphics, cognitive science, and literary theory, I offer ways to think about images (Chandra et al., 2022) and stories (Chandra et al., 2023a, 2023b) as objects to be designed with respect to a model of the audience. In particular, if we think of the audience's mind as solving inverse problems—perception as inverse rendering, action understanding as inverse planning—then we can think of expression as solving a kind of *inverse* inverse problem.

I then show how to implement such "inverse inverse" methods computationally. Starting with classic Bayesian models of vision and social cognition, I present algorithms for optimizing over inference to create "adversarial examples" that evoke various desired inferences in those models. For example, we optimize images that are visual illusions (Chapter 1), and animations that tell surprising stories (Chapter 2). Because the Bayesian models capture human intuitions well (better than, say, typical neural network models), the optimized stimuli "transfer" to evoke similar experiences in humans. I demonstrate this with a variety of human subject studies.