Repository logo
Log in(current)
Repository logoMIT Open ScholarshipDSpace@MIT
  1. Home
  2. Center for Brains, Minds & Machines
  3. Publications
  4. CBMM Memo Series
  5. Spatiotemporal interpretation features in the recognition of dynamic images

Spatiotemporal interpretation features in the recognition of dynamic images

Thumbnail Image
Download
Name

CBMM-Memo-094.pdf

Size

1.21 MB

Format

Adobe PDF

Checksum (MD5)

efc36f79f7228b7ea75b38588632bcca

Author(s)
Ben-Yosef, Guy
•
Kreiman, Gabriel
•
Ullman, Shimon
Date Issued
November 21, 2018
Publisher
Center for Brains, Minds and Machines (CBMM)
Series/Report no.
CBMM Memo Series;094
Abstract
Objects and their parts can be visually recognized and localized from purely spatial information in static images and also from purely temporal information as in the perception of biological motion. Cortical regions have been identified, which appear to specialize in visual recognition based on either static or dynamic cues, but the mechanisms by which spatial and temporal information is integrated is only poorly understood. Here we show that visual recognition of objects and actions can be achieved by efficiently combining spatial and motion cues in configurations where each source on its own is insufficient for recognition. This analysis is obtained by the identification of minimal spatiotemporal configurations: these are short videos in which objects and their parts, along with an action being performed, can be reliably recognized, but any reduction in either space or time makes them unrecognizable. State-of-the-art computational models for recognition from dynamic images based on deep 2D and 3D convolutional networks cannot replicate human recognition in these configurations. Action recognition in minimal spatiotemporal configurations is invariably accompanied by full human interpretation of the internal components of the image and their inter-relations. We hypothesize that this gap is due to mechanisms for full spatiotemporal interpretation process, which in human vision is an integral part of recognizing dynamic event, but is not sufficiently represented in current DNNs.
Persistent DSpace Link
http://hdl.handle.net/1721.1/119248
Repository logo
PrivacyPermissionsAccessibilityContact us
Repository logo
Notify us about copyright concerns.