Personalized Modeling of Real-World Vocalizations from Nonverbal Individuals
Name
3382507.3418854.pdf
Description
Published version
Size
6.63 MB
Format
Adobe PDF
Checksum (MD5)
5a85d1df039837baf0e78ea98ce9aca7
Author(s)
Narain, J
Johnson, KT
Ferguson, C
O'Brien, A
Talkar, T
Weninger, YZ
Wofford, P
Quatieri, T
Picard, Rosalind W.
Maes, P
Date Issued
2020
Journal
ICMI 2020 - Proceedings of the 2020 International Conference on Multimodal Interaction
Publisher
ACM
Citation
Narain, J, Johnson, KT, Ferguson, C, O'Brien, A, Talkar, T et al. 2020. "Personalized Modeling of Real-World Vocalizations from Nonverbal Individuals." ICMI 2020 - Proceedings of the 2020 International Conference on Multimodal Interaction.
Version
Final published version
Abstract
© 2020 Owner/Author. Nonverbal vocalizations contain important affective and communicative information, especially for those who do not use traditional speech, including individuals who have autism and are non- or minimally verbal (nv/mv). Although these vocalizations are often understood by those who know them well, they can be challenging to understand for the community-at-large. This work presents (1) a methodology for collecting spontaneous vocalizations from nv/mv individuals in natural environments, with no researcher present, and personalized in-the-moment labels from a family member; (2) speaker-dependent classification of these real-world sounds for three nv/mv individuals; and (3) an interactive application to translate the nonverbal vocalizations in real time. Using support-vector machine and random forest models, we achieved speaker-dependent unweighted average recalls (UARs) of 0.75, 0.53, and 0.79 for the three individuals, respectively, with each model discriminating between 5 nonverbal vocalization classes. We also present first results for real-time binary classification of positive- and negative-affect nonverbal vocalizations, trained using a commercial wearable microphone and tested in real time using a smartphone. This work informs personalized machine learning methods for non-traditional communicators and advances real-world interactive augmentative technology for an underserved population.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Lincoln Laboratory
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3382507.3418854
