Self-Supervised Audio-Visual Co-Segmentation
Name
1904.09013.pdf
Description
Accepted version
Size
3.45 MB
Format
Unknown
Checksum (MD5)
ce4805f5d96afcca47ad24252c32bc46
Author(s)
Rouditchenko, Andrew
Zhao, Hang
Gan, Chuang
McDermott, Joshua Hartman
Torralba, Antonio
Date Issued
May 2019
Publisher
IEEE
Citation
Rouditchenko, Andrew, Zhao, Hang, Gan, Chuang, McDermott, Josh and Torralba, Antonio. 2019. "Self-Supervised Audio-Visual Co-Segmentation."
Version
Author's final manuscript
Abstract
© 2019 IEEE. Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this paper we develop a neural network model for visual object segmentation and sound source separation that learns from natural videos through self-supervision. The model is an extension of recently proposed work that maps image pixels to sounds [1]. Here, we introduce a learning approach to disentangle concepts in the neural networks, and assign semantic categories to network feature channels to enable independent image segmentation and sound source separation after audio-visual training on videos. Our evaluations show that the disentangled model outperforms several baselines in semantic segmentation and sound source separation.
MIT Department
MIT-IBM Watson AI Lab
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/icassp.2019.8682467
