Unsupervised audio-visual learning in the wild
Name
1192539279-MIT.pdf
Size
15.01 MB
Format
Adobe PDF
Checksum (MD5)
56dd9004d5633c45d139d9e912581a79
Author(s)
Boggust, Angie Wynne.
Advisor(s)
James Glass and David Harwath.
Date Issued
2020
Publisher
Massachusetts Institute of Technology
Abstract
Human babies posses the innate ability to learn language using correspondences between what they see and what they hear. Yet, current unsupervised methods for learning visually grounded language often rely on time-consuming and expensive data annotation, such as spoken language captions of images or textual summaries of video. In this thesis, we eliminate the need for annotation by learning an audio-visual grounding between instructional videos and their audio waveforms. We present two methods capable of learning a joint audio-visual embedding space from video input. In the first method, we apply the DAVEnet model architecture (Harwath et al., 2016) to visual frames and audio segments extracted from over 3000 instructional cooking videos. In the second method, we introduce the Video DAVEnet architecture -- an unsupervised network that learns a joint audio-visual embedding space from raw video -- and apply it to 1.2 million publicly available instructional YouTube videos. While the methods we compare to learn from video and human-generated textual summaries, our methods achieve state of the art performance on downstream audio and visual recall tasks using only raw video data. Finally, we perform analysis of the learned audio-visual embedding space and show that our models learn salient audio-visual concepts, such as "oil", "onion", and "fry" when applied to cooking videos from the YouCook2 dataset (Zhou et al., 2018a).
Description
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, May, 2020
Cataloged from the official PDF of thesis.
Includes bibliographical references (pages 71-77).
Subjects
Electrical Engineering and Computer Science.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided.
Persistent DSpace Link
