MIT Libraries logoDSpace@MIT

MIT
View Item 
  • DSpace@MIT Home
  • MIT Libraries
  • MIT Theses
  • Graduate Theses
  • View Item
  • DSpace@MIT Home
  • MIT Libraries
  • MIT Theses
  • Graduate Theses
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

Unsupervised audio-visual learning in the wild

Author(s)
Boggust, Angie Wynne.
Thumbnail
Download1192539279-MIT.pdf (15.01Mb)
Other Contributors
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science.
Advisor
James Glass and David Harwath.
Terms of use
MIT theses may be protected by copyright. Please reuse MIT thesis content according to the MIT Libraries Permissions Policy, which is available through the URL provided. http://dspace.mit.edu/handle/1721.1/7582
Metadata
Show full item record
Abstract
Human babies posses the innate ability to learn language using correspondences between what they see and what they hear. Yet, current unsupervised methods for learning visually grounded language often rely on time-consuming and expensive data annotation, such as spoken language captions of images or textual summaries of video. In this thesis, we eliminate the need for annotation by learning an audio-visual grounding between instructional videos and their audio waveforms. We present two methods capable of learning a joint audio-visual embedding space from video input. In the first method, we apply the DAVEnet model architecture (Harwath et al., 2016) to visual frames and audio segments extracted from over 3000 instructional cooking videos. In the second method, we introduce the Video DAVEnet architecture -- an unsupervised network that learns a joint audio-visual embedding space from raw video -- and apply it to 1.2 million publicly available instructional YouTube videos. While the methods we compare to learn from video and human-generated textual summaries, our methods achieve state of the art performance on downstream audio and visual recall tasks using only raw video data. Finally, we perform analysis of the learned audio-visual embedding space and show that our models learn salient audio-visual concepts, such as "oil", "onion", and "fry" when applied to cooking videos from the YouCook2 dataset (Zhou et al., 2018a).
Description
Thesis: M. Eng., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, May, 2020
 
Cataloged from the official PDF of thesis.
 
Includes bibliographical references (pages 71-77).
 
Date issued
2020
URI
https://hdl.handle.net/1721.1/127377
Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Publisher
Massachusetts Institute of Technology
Keywords
Electrical Engineering and Computer Science.

Collections
  • Graduate Theses

Browse

All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login

Statistics

OA StatisticsStatistics by CountryStatistics by Department
MIT Libraries
PrivacyPermissionsAccessibilityContact us
MIT
Content created by the MIT Libraries, CC BY-NC unless otherwise noted. Notify us about copyright concerns.