Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
Name
1712.07271.pdf
Description
Submitted version
Size
8.82 MB
Format
Adobe PDF
Checksum (MD5)
c3c3ee4f1d98876bb56f4e058b8ea7f9
Author(s)
Owens, Andrew
Wu, Jiajun
McDermott, Josh H
Freeman, William T
Torralba, Antonio
Date Issued
2018
Journal
International Journal of Computer Vision
Publisher
Springer Nature America, Inc
Version
Original manuscript
Abstract
© 2018, Springer Science+Business Media, LLC, part of Springer Nature. The sound of crashing waves, the roar of fast-moving cars—sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds can be used as a supervisory signal for learning visual models. To demonstrate this, we train a convolutional neural network to predict a statistical summary of the sound associated with a video frame. We show that, through this process, the network learns a representation that conveys information about objects and scenes. We evaluate this representation on several recognition tasks, finding that its performance is comparable to that of other state-of-the-art unsupervised learning methods. Finally, we show through visualizations that the network learns units that are selective to objects that are often associated with characteristic sounds. This paper extends an earlier conference paper, Owens et al. (in: European conference on computer vision, 2016b), with additional experiments and discussion.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
McGovern Institute for Brain Research at MIT
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/S11263-018-1083-5
