Visually Indicated Sounds
Name
Visually indicated sounds.pdf
Size
6.7 MB
Format
Adobe PDF
Checksum (MD5)
db80d22db22a45030233958c47ec31de
Author(s)
Isola, Phillip
McDermott, Josh
Adelson, Edward H.
Freeman, William T.
Torralba, Antonio
Owens, Andrew Hale
Date Issued
June 2016
Journal
IEEE Conference on Computer Vision and Pattern Recognition, 2016. CVPR 2016
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Owens, Andrew, Phillip Isola, Josh McDermott, Antonio Torralba, Edward H. Adelson, and William T. Freeman. “Visually Indicated Sounds.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2016). © 2016 Institute of Electrical and Electronics Engineers (IEEE)
Version
Original manuscript
Abstract
Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this paper, we propose the task of predicting what sound an object makes when struck as a way of studying physical interactions within a visual scene. We present an algorithm that synthesizes sound from silent videos of people hitting and scratching objects with a drumstick. This algorithm uses a recurrent neural network to predict sound features from videos and then produces a waveform from these features with an example-based synthesis procedure. We show that the sounds predicted by our model are realistic enough to fool participants in a "real or fake" psychophysical experiment, and that they convey significant information about material properties and physical interactions.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/CVPR.2016.264
