EgoCom: A Multi-person Multi-modal Egocentric Communications Dataset
Name
09200754.pdf
Size
4.46 MB
Format
Adobe PDF
Checksum (MD5)
a6dee5c320f6a47fad06edc934fc9690
Author(s)
Northcutt, Curtis George
Zha, Shengxin
Lovegrove, Steven
Newcombe, Richard
Date Issued
September 2020
Journal
IEEE Transactions on Pattern Analysis and Machine Intelligence
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Northcutt, Curtis G. et al. "EgoCom: A Multi-person Multi-modal Egocentric Communications Dataset." Forthcoming in IEEE Transactions on Pattern Analysis and Machine Intelligence (September 2020): dx.doi.org/10.1109/tpami.2020.3025105.
Version
Final published version
Abstract
Multi-modal datasets in artificial intelligence (AI) often capture a third-person perspective, but our embodied human intelligence evolved with sensory input from the egocentric, first-person perspective. Towards embodied AI, we introduce the Egocentric Communications (EgoCom) dataset to advance the state-of-the-art in conversational AI, natural language, audio speech analysis, computer vision, and machine learning. EgoCom is a first-of-its-kind natural conversations dataset containing multi-modal human communication data captured simultaneously from the participants' egocentric perspectives. EgoCom includes 38.5 hours of synchronized embodied stereo audio, egocentric video with 240,000 ground-truth, time-stamped word-level transcriptions and speaker labels from 34 diverse speakers. We study baseline performance on two novel applications that benefit from embodied data: (1) predicting turn-taking in conversations and (2) multi-speaker transcription. For (1), we investigate Bayesian baselines to predict turn-taking within 5% of human performance. For (2), we use simultaneous egocentric capture to combine Google speech-to-text outputs, improving global transcription by 79% relative to a single perspective. Both applications exploit EgoCom's synchronous multi-perspective data to augment performance of embodied AI tasks.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/tpami.2020.3025105
