Interfaces for exploring human memorability and cognition

Abstract

Compared to images, videos are more dynamic, complex, and diverse. Analyzing and understanding video content is a tough cognitive challenge for computer vision. These challenges include analyzing the memorability of the video, determining semantic abstractions, and detecting when videos are fake. To better understand these challenges, we look to analyzing human behavior through the use of multiple interfaces. These interfaces are games designed to collect semantic and behavioral information from humans via Amazon Mechanical Turk. We process and analyze the data to incorporate into our models and provide a human baseline for comparison. We find that incorporating semantic attributes improves the capabilities of predicting the memorability of the video, and that our models are able to perform cognitive tasks related to semantic relational abstractions with near human accuracy.