MIT Libraries homeMIT Libraries logoDSpace@MIT

MIT
View Item 
  • DSpace@MIT Home
  • MIT Open Access Articles
  • MIT Open Access Articles
  • View Item
  • DSpace@MIT Home
  • MIT Open Access Articles
  • MIT Open Access Articles
  • View Item
JavaScript is disabled for your browser. Some features of this site may not work without it.

Real-Time Inference of Mental States from Facial Expressions and Upper Body Gestures

Author(s)
Baltrusaitis, Tadas; McDuff, Daniel Jonathan; Banda, Ntombikayise; Mahmoud, Marwa; el Kaliouby, Rana; Robinson, Peter; Picard, Rosalind W.; ... Show more Show less
Thumbnail
DownloadFERA 2011_McDuff.pdf (300.0Kb)
OPEN_ACCESS_POLICY

Open Access Policy

Creative Commons Attribution-Noncommercial-Share Alike

Terms of use
Creative Commons Attribution-Noncommercial-Share Alike 3.0 http://creativecommons.org/licenses/by-nc-sa/3.0/
Metadata
Show full item record
Abstract
We present a real-time system for detecting facial action units and inferring emotional states from head and shoulder gestures and facial expressions. The dynamic system uses three levels of inference on progressively longer time scales. Firstly, facial action units and head orientation are identified from 22 feature points and Gabor filters. Secondly, Hidden Markov Models are used to classify sequences of actions into head and shoulder gestures. Finally, a multi level Dynamic Bayesian Network is used to model the unfolding emotional state based on probabilities of different gestures. The most probable state over a given video clip is chosen as the label for that clip. The average F1 score for 12 action units (AUs 1, 2, 4, 6, 7, 10, 12, 15, 17, 18, 25, 26), labelled on a frame by frame basis, was 0.461. The average classification rate for five emotional states (anger, fear, joy, relief, sadness) was 0.440. Sadness had the greatest rate, 0.64, anger the smallest, 0.11.
Date issued
2011-03
URI
http://hdl.handle.net/1721.1/67458
Department
Massachusetts Institute of Technology. Media Laboratory
Journal
2011 IEEE International Conference on Automatic Face & Gesture Recognition and Workshops (FG 2011)
Publisher
Institute of Electrical and Electronics Engineers
Citation
Baltrusaitis, Tadas et al. “Real-time Inference of Mental States from Facial Expressions and Upper Body Gestures.” Face and Gesture 2011. Santa Barbara, CA, USA, 2011. 909-914.
Version: Author's final manuscript
ISBN
978-1-4244-9140-7

Collections
  • MIT Open Access Articles

Browse

All of DSpaceCommunities & CollectionsBy Issue DateAuthorsTitlesSubjectsThis CollectionBy Issue DateAuthorsTitlesSubjects

My Account

Login

Statistics

OA StatisticsStatistics by CountryStatistics by Department
MIT Libraries homeMIT Libraries logo

Find us on

Twitter Facebook Instagram YouTube RSS

MIT Libraries navigation

SearchHours & locationsBorrow & requestResearch supportAbout us
PrivacyPermissionsAccessibility
MIT
Massachusetts Institute of Technology
Content created by the MIT Libraries, CC BY-NC unless otherwise noted. Notify us about copyright concerns.