Unsupervised audio-visual learning in the wild

Abstract

Human babies posses the innate ability to learn language using correspondences between what they see and what they hear. Yet, current unsupervised methods for learning visually grounded language often rely on time-consuming and expensive data annotation, such as spoken language captions of images or textual summaries of video. In this thesis, we eliminate the need for annotation by learning an audio-visual grounding between instructional videos and their audio waveforms. We present two methods capable of learning a joint audio-visual embedding space from video input. In the first method, we apply the DAVEnet model architecture (Harwath et al., 2016) to visual frames and audio segments extracted from over 3000 instructional cooking videos. In the second method, we introduce the Video DAVEnet architecture -- an unsupervised network that learns a joint audio-visual embedding space from raw video -- and apply it to 1.2 million publicly available instructional YouTube videos. While the methods we compare to learn from video and human-generated textual summaries, our methods achieve state of the art performance on downstream audio and visual recall tasks using only raw video data. Finally, we perform analysis of the learned audio-visual embedding space and show that our models learn salient audio-visual concepts, such as "oil", "onion", and "fry" when applied to cooking videos from the YouCook2 dataset (Zhou et al., 2018a).