Real-Time Inference of Mental States from Facial Expressions and Upper Body Gestures
Name
FERA 2011_McDuff.pdf
Size
300.06 KB
Format
Adobe PDF
Checksum (MD5)
de31d169a10860b643f9571d1832fec8
Author(s)
Baltrusaitis, Tadas
McDuff, Daniel Jonathan
Banda, Ntombikayise
Mahmoud, Marwa
el Kaliouby, Rana
Robinson, Peter
Picard, Rosalind W.
Date Issued
March 2011
Journal
2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)
Publisher
Institute of Electrical and Electronics Engineers
Citation
Baltrusaitis, Tadas et al. “Real-time Inference of Mental States from Facial Expressions and Upper Body Gestures.” Face and Gesture 2011. Santa Barbara, CA, USA, 2011. 909-914.
Version
Author's final manuscript
Abstract
We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike 3.0
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/FG.2011.5771372
