Temporal Relational Reasoning in Videos
Name
1711.08496.pdf
Description
Accepted version
Size
4.04 MB
Format
Adobe PDF
Checksum (MD5)
7496334cb33ff712566399f980a0a21d
Author(s)
Zhou, Bolei
Andonian, Alexander Joseph
Oliva, Aude
Torralba, Antonio
Date Issued
October 2018
Journal
Computer Vision - EECV 2018
Publisher
Springer International Publishing
Citation
Zhou, Bolei, et al. "Temporal Relational Reasoning in Videos." European Conference on Computer Vision, 2018, Munich, Germany
Version
Author's final manuscript
Abstract
Temporal relational reasoning, the ability to link meaningful transformations of objects or entities over time, is a fundamental property of intelligent species. In this paper, we introduce an effective and interpretable network module, the Temporal Relation Network (TRN), designed to learn and reason about temporal dependencies between video frames at multiple time scales. We evaluate TRN-equipped networks on activity recognition tasks using three recent video datasets - Something-Something, Jester, and Charades - which fundamentally depend on temporal relational reasoning. Our results demonstrate that the proposed TRN gives convolutional neural networks a remarkable capacity to discover temporal relations in videos. Through only sparsely sampled video frames, TRN-equipped networks can accurately predict human-object interactions in the Something-Something dataset and identify various human gestures on the Jester dataset with very competitive performance. TRN-equipped networks also outperform two-stream networks and 3D convolution networks in recognizing daily activities in the Charades dataset. Further analyses show that the models learn intuitive and interpretable visual common sense knowledge in videos (Code and models are available at http://relation.csail.mit.edu/.).
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1007/978-3-030-01246-5_49
