Online optimization with energy based models

Abstract

This thesis examines the power of applying optimization on learned neural networks, referred to as Energy Based Models (EBMs). We first present methods that enable scalable training of EBMs, allowing an optimization procedure to generate high resolution images. We simultaneously show that resultant models are robust, compositional, and are further easy to learn online. Next we showcase how this optimization procedure can also be used to formulate plans in interactive environments. We further showcase how a similar procedure can be used to learn neural energy functions for proteins, enabling structural recovery through optimization. Finally, we show that by defining generation as a optimization procedure, we can combine generative models from different domains together, and apply optimization on the joint model. We show that this allows us to apply various logical operations on images generation, as well as learn to generate new concepts in a continual manner.