The Sound of Motions
Name
1904.05979.pdf
Description
Submitted version
Size
5.93 MB
Format
Adobe PDF
Checksum (MD5)
9db43760c43feaa917662a802ada325a
Author(s)
Zhao, Hang
Gan, Chuang
Ma, Wei-Chiu
Torralba, Antonio
Date Issued
October 2019
Journal
Proceedings of the IEEE International Conference on Computer Vision
Publisher
IEEE
Citation
Zhao, Hang, Gan, Chuang, Ma, Wei-Chiu and Torralba, Antonio. 2019. "The Sound of Motions." Proceedings of the IEEE International Conference on Computer Vision, 2019-October.
Version
Original manuscript
Abstract
© 2019 IEEE. Sounds originate from object motions and vibrations of surrounding air. Inspired by the fact that humans is capable of interpreting sound sources from how objects move visually, we propose a novel system that explicitly captures such motion cues for the task of sound localization and separation. Our system is composed of an end-to-end learnable model called Deep Dense Trajectory (DDT), and a curriculum learning scheme. It exploits the inherent coherence of audio-visual signals from a large quantities of unlabeled videos. Quantitative and qualitative evaluations show that comparing to previous models that rely on visual appearance cues, our motion based system improves performance in separating musical instrument sounds. Furthermore, it separates sound components from duets of the same category of instruments, a challenging problem that has not been addressed before.
MIT Department
MIT-IBM Watson AI Lab
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/iccv.2019.00182
