Real-Time Inference of Mental States from Facial Expressions and Upper Body Gestures
Author(s)Baltrusaitis, Tadas; McDuff, Daniel Jonathan; Banda, Ntombikayise; Mahmoud, Marwa; el Kaliouby, Rana; Robinson, Peter; Picard, Rosalind W.; ... Show more Show less
MetadataShow full item record
We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.
DepartmentMassachusetts Institute of Technology. Media Laboratory
2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)
Institute of Electrical and Electronics Engineers
Baltrusaitis, Tadas et al. “Real-time Inference of Mental States from Facial Expressions and Upper Body Gestures.” Face and Gesture 2011. Santa Barbara, CA, USA, 2011. 909-914.
Author's final manuscript