Neural techniques for modeling visually grounded speech

Abstract

In this thesis, I explore state of the art techniques for using neural networks to learn semantically-rich representations for visual and audio data. In particular, I analyze and extend the model introduced by Harwath et al. (2016), a neural architecture which learns a non-linear similarity metric between images and audio captions using sampled margin rank loss. In Chapter 1, I provide a background on multimodal learning and motivate the need for further research in the area. In addition, I give an overview of Harwath et al. (2016)'s model, variants of which will be used throughout the rest of the thesis. In Chapter 2, I present a quantitative and qualitative analysis of the modality retrieval behavior of the state of the art architecture used by Harwath et al. (2016), identifying a bias towards certain examples and proposing a solution to counteract that bias. In Chapter 3, I introduce the property of modality invariance and explain a regularization technique I created to promote this property in learned semantic embedding spaces. In Chapter 4, I apply the architecture to a new dataset containing videos, which offers unique opportunities to include temporal visual data and ambient audio unavailable in images. In addition, the video domain presents new challenges, as the data density increases with the additional time dimension. I conclude with a discussion about multimodal learning, language acquisition, and unsupervised learning in general.