Holistic Affect Recognition Using PaNDA: Paralinguistic Non-metric Dimensional Analysis
Name
Zhang19_HAR.pdf
Size
9.76 MB
Format
Adobe PDF
Checksum (MD5)
dd41915b1a4ce7282a0335ac31584009
Author(s)
Zhang, Yue
Weninger, Felix
Bjorn, Schuller
Picard, Rosalind W.
Date Issued
December 2019
Journal
IEEE Transactions on Affective Computing
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Zhang, Y. et al. "Holistic Affect Recognition Using PaNDA: Paralinguistic Non-metric Dimensional Analysis," IEEE Transactions on Affective Computing (December 2019). © 2019 IEEE
Version
Final published version
Abstract
Humans perceive emotion from each other using a holistic perspective, accounting for diverse personal, non-emotional variables that shape expression. In contrast, today's algorithms are mainly designed to recognize emotion in isolation. In this work, we propose a multi-task learning approach to jointly learn the recognition of affective states from speech along with various speaker attributes. A problem with multi-task learning is that sometimes inductive transfer can negatively impact performance. To mitigate negative transfer, we introduce the Paralinguistic Non-metric Dimensional Analysis (PaNDA) method that systematically measures task relatedness and also enables visualizing the topology of affective phenomena as a whole. In addition, we present a generic framework that conflates the concepts of single-task and multi-task learning. Using this framework, we construct two models that demonstrate holistic affect recognition: one treats all tasks as equally related, whereas the other one incorporates the task correlations between a main task and its supporting tasks obtained from PaNDA. Both models employ a multi-task deep neural network, in which separate output layers are used to predict discrete and continuous attributes, while hidden layers are shared across different tasks. On average across 18 classification and regression tasks, the weighted multi-task learning with PaNDA significantly improves performance compared to single-task and unweighted multi-task learning.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/taffc.2019.2961881
