Dyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset
Name
3382507.3418842.pdf
Description
Published version
Size
2.78 MB
Format
Adobe PDF
Checksum (MD5)
b1368a2a9e5931ae27b93df4cfe75dee
Author(s)
Chen, H
Zhang, Y
Weninger, F
Picard, Rosalind W.
Breazeal, C
Park, HW
Date Issued
2020
Journal
ICMI 2020 - Proceedings of the 2020 International Conference on Multimodal Interaction
Publisher
ACM
Citation
Chen, H, Zhang, Y, Weninger, F, Picard, R, Breazeal, C et al. 2020. "Dyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset." ICMI 2020 - Proceedings of the 2020 International Conference on Multimodal Interaction.
Version
Final published version
Abstract
© 2020 Owner/Author. Automatic speech-based affect recognition of individuals in dyadic conversation is a challenging task, in part because of its heavy reliance on manual pre-processing. Traditional approaches frequently require hand-crafted speech features and segmentation of speaker turns. In this work, we design end-to-end deep learning methods to recognize each person's affective expression in an audio stream with two speakers, automatically discovering features and time regions relevant to the target speaker's affect. We integrate a local attention mechanism into the end-to-end architecture and compare the performance of three attention implementations - one mean pooling and two weighted pooling methods. Our results show that the proposed weighted-pooling attention solutions are able to learn to focus on the regions containing target speaker's affective information and successfully extract the individual's valence and arousal intensity. Here we introduce and use a "dyadic affect in multimodal interaction - parent to child"(DAMI-P2C) dataset collected in a study of 34 families, where a parent and a child (3-7 years old) engage in reading storybooks together. In contrast to existing public datasets for affect recognition, each instance for both speakers in the DAMI-P2C dataset is annotated for the perceived affect by three labelers. To encourage more research on the challenging task of multi-speaker affect sensing, we make the annotated DAMI-P2C dataset publicly available, including acoustic features of the dyads' raw audios, affect annotations, and a diverse set of developmental, social, and demographic profiles of each dyad.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3382507.3418842
