Joint generation of image and text with GANs

Abstract

The computer vision and natural language processing communities have come together on image captioning related problems, but the fields have remained largely disjoint. There has been work in transforming text to text, images to text, text to images, and images to images, and work on generating images from nothing, and text from nothing, but no work on generating images and text together. This work looks at using GAN methods in order to generate images and text simultaneously using a shared representation. Visual representations of text are employed to use GAN techniques for both images and text. We present a framework for jointly generating images and text with a similarity loss that allows the model to learn a semantic representation.