Neural mechanisms underlying core visual perception of objects

Abstract

Visual perception of objects is a computationally challenging problem and fundamental to human well-being. Extensive previous research has revealed that the inferior temporal cortex (IT), a high-level visual area, is involved in various aspects of visual perception. Yet, little is known about: how IT neural responses to objects support human perception of the objects; and how IT responses are produced from retinal images of objects. The goal of this research is to tackle these two related questions and find out explicit, quantitative mechanisms that describe human core visual perception of objects, a remarkable ability achieved with brief (<200ms) image viewing duration. We first operationally define the core visual perception by measuring behavioral reports of human subjects in hundreds of visual tasks. These tasks are designed to systematically assess subjects' ability to estimate key visual parameters of an object in an image, such as the object's category, identity, position, size, and viewpoint angles. Combined with a rich dataset of monkey visual neural responses to the same task images, we systematically explore a large number of explicit hypotheses that might explain the human behavioral reports. Here, we demonstrate that weighted linear sums of IT responses robustly predict the human pattern of behavior. Moreover, we show that performance-optimized hierarchical neural networks explain a large portion of neural responses of high-level visual areas including IT. These results establish a working mechanistic model of core visual perception by providing an end-to-end understanding of the human visual system from images to neural responses to behavior.