Predicting Motivations of Actions by Leveraging Text
Name
Torralba_Predicting motivations.pdf
Size
11.89 MB
Format
Adobe PDF
Checksum (MD5)
112cd5083ed19c7cff9d20fe75806669
Author(s)
Pirsiavash, Hamed
Vondrick, Carl Martin
Oktay, Deniz
Torralba, Antonio
Date Issued
December 2016
Journal
2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Publisher
Institute of Electrical and Electronics Engineers (IEEE)
Citation
Vondrick, Carl, Deniz Oktay, Hamed Pirsiavash, and Antonio Torralba. “Predicting Motivations of Actions by Leveraging Text.” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 27-30 June 2016, Las Vegas, Nevada, IEEE, 2016.
Version
Author's final manuscript
Abstract
Understanding human actions is a key problem in computer vision. However, recognizing actions is only the first step of understanding what a person is doing. In this paper, we introduce the problem of predicting why a person has performed an action in images. This problem has many applications in human activity understanding, such as anticipating or explaining an action. To study this problem, we introduce a new dataset of people performing actions annotated with likely motivations. However, the information in an image alone may not be sufficient to automatically solve this task. Since humans can rely on their lifetime of experiences to infer motivation, we propose to give computer vision systems access to some of these experiences by using recently developed natural language models to mine knowledge stored in massive amounts of text. While we are still far away from fully understanding motivation, our results suggest that transferring knowledge from language into vision can help machines understand why people in images might be performing an action.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1109/CVPR.2016.327
