Learning Generalizable Systems by Learning Composable Energy Landscapes

Abstract

How can we construct intelligent embodied agents in the physical world? Such agents should be able to autonomously solve tasks that have not been seen before, subject to external disturbances in the environment, as well as new combinations of factors such as lighting, varying sensor inputs, and unexpected interactions with agents and other objects. An important subgoal towards constructing such intelligent agents is to construct models that can robustly generalize, not only to distributions of tasks similar to ones seen at training time but also to new unseen distributions. This departs from standard machine learning techniques which usually assume identical training and test distributions. Towards this goal, in this dissertation, we’ll illustrate how we can achieve certain forms of generalization by estimating energy landscapes over possible predictions for each task, with accurate predictions assigned lower energy. This modeling choice formulates prediction as a search process on the energy landscape, enabling zero-shot generalization to new constraints by adapting the energy landscape. In addition, this allows us to generalize to entirely new distributions of tasks in a zero-shot manner by composing multiple learned energy landscapes together. In this dissertation, we first introduce a set of techniques to train energy landscapes and an algebra in which we can compose and discover composable energy landscapes. Next, we illustrate how energy landscapes can be composed in a diverse set of ways, ranging from logical operators, probability distributions, graphical models, constraints, and hierarchical compositions, enabling effective generalization across vision, decision-making, multimodal, and scientific settings.