Characterizing Image Recognition Difficulty in Artificial and Biological Visual Processing

Abstract

In recent years, computational models trained to do object recognition have become increasingly capable. Models have demonstrated significant improvements and have achieved saturated performance on many standard image classification benchmarks sparking discussion of whether these models have achieved parity with human object recognition ability and whether we can consider this problem solved. However, these models continue to fail in real-world applications and in un-human-like ways creating a disparity between the performance that benchmarks report and the performance that users experience. In this thesis, we investigate why standard datasets are misaligned with real-world performance by exploring image recognition difficulty as defined by human psychophysics. Using behavioral experiments with humans, we are able to identify images that humans struggle to recognize and investigate the prevalence of these images in datasets and their effect on model performance. To shed light on how humans are able to recognize these images, we conduct preliminary analysis with neuroimaging to take the first steps at identifying the neural signature of image difficulty.