Understanding object recognition performance at scale in machines and humans

Abstract

Machine performance on the object classication and detection tasks is remark- ably high today. On some datasets, such as ImageNet, it seems to surpass human performance according to recently published results. Yet when we run detectors over real videos we observe that machine performance is far inferior to human performance. We aim to resolve this disconnect and understand the true state of machine and human performance for object recognition. To do this we have gathered a new large image dataset, via the use of Amazon Mechanical Turk, with novel methodology and evaluation mechanisms to both answer questions about how well humans recognize objects and to carefully characterize machine performance. We have found that the performance of current state-of-the-art object detectors drops significantly when run on our dataset: from 71% accuracy to 25% accuracy accuracy. This drop in performance indicates that object detection is not a solved problem, despite previous benchmarks.