CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism
Author(s)
Rudovic, Ognjen; Utsumi, Yuria; Lee, Jaeryoung; Hernandez, Javier; Ferrer, Eduardo Castello; Schuller, Bjorn; Picard, Rosalind W.; ... Show more Show less
DownloadAccepted version (3.802Mb)
Terms of use
Metadata
Show full item recordAbstract
© 2018 IEEE. Many children on autism spectrum have atypical behavioral expressions of engagement compared to their neu-rotypical peers. In this paper, we investigate the performance of deep learning models in the task of automated engagement estimation from face images of children with autism. Specifically, we use the video data of 30 children with different cultural backgrounds (Asia vs. Europe) recorded during a single session of a robot-assisted autism therapy. We perform a thorough evaluation of the proposed deep architectures for the target task, including within- and across-culture evaluations, as well as when using the child-independent and child-dependent settings. We also introduce a novel deep learning model, named CultureNet, which efficiently leverages the multi-cultural data when performing the adaptation of the proposed deep architecture to the target culture and child. We show that due to the highly heterogeneous nature of the image data of children with autism, the child-independent models lead to overall poor estimation of target engagement levels. On the other hand, when a small amount of data of target children is used to enhance the model learning, the estimation performance on the held-out data from those children increases significantly. This is the first time that the effects of individual and cultural differences in children with autism have empirically been studied in the context of deep learning performed directly from face images.
Date issued
2018-10Department
Massachusetts Institute of Technology. Media Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer SciencePublisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Rudovic, Ognjen, Utsumi, Yuria, Lee, Jaeryoung, Hernandez, Javier, Ferrer, Eduardo Castello et al. 2018. "CultureNet: A Deep Learning Approach for Engagement Intensity Estimation from Face Images of Children with Autism."
Version: Author's final manuscript