Towards abstractive captioning of infographics

Abstract

Machine understanding of text-based narratives have predominantly focused on documents with rigid hierarchical structures and sequentially ordered inputs. These inputs include documents such as news stories, encyclopedia entries, books, and many others. However, little research has focused on understanding text-based information without this structure. Current text understanding models fail when information is presented in less structured ways, without a clear and pre-defined spatial arrangement of the content. This thesis explores a subset of components required for understanding infographics -- documents whose structure is not necessarily linear and whose content may involve a variety of images. We expand on state-of-the-art methodologies in character recognition and text summarization in order to better understand how to process content without a pre-determined spatial arrangement, and subsequently generate captions for given infographics automatically. To shine light at the reasoning behind the captions being generated, we develop a graphical user interface that helps visualize the portions of a document being used when generating specific parts of a caption.