Emergent Capabilities of Generative Models: “Software 3.0” and Beyond

Abstract

Modern AI algorithms are rapidly becoming ubiquitous in everyday life and have even been touted as the new “Software 2.0” stack by prominent researchers in the field. Indeed, these algorithms are fundamentally changing the way we interact with, potentially even how we will program computers to achieve desired outcomes. In this thesis, we advocate that wielding control over these increasingly powerful models is important for progression in the field, and more importantly, for ensuring that models deployed in the real world behave in the ways we would like and preventing cases where they may do unintended harm. First, we present an empirical study in which we train a large-scale Generative Adversarial Network (GAN) on the MIT Places365 dataset, achieving state-of-the-art Inception scores and Fréchet Inception distance, metrics that are used to evaluate image synthesis quality. We then introduce a GAN framework, GANalyze, that allows one to make targeted manipulations to various cognitive attributes of GAN generated imagery, such as memorability and emotional valence, and use this framework to surface “visual definitions” of these properties. Through behavioral experiments, we verify that our method discovers image manipulations that causally affect human memory performance. Finally, we build on this framework by incorporating a powerful new pretrained text-image semantic similarity model to create a novel image editing application that allows users to “paint by word.” All together, this progression of work underscores the advantages of the emerging “Software 3.0” stack, whereby programmers are tasked with orchestrating and finetuning the interactions between large-scale foundation models to carry out higher-order tasks.