Human-computer perception : modeling visual perceptual attributes

Abstract

Human perception provides clues as to which visual content is most crucial or engaging. Where people look indicates what they pay attention to and find relevant; what people remember is what the human brain deems to be worthy of preservation. Recently, Deep Neural Networks have made it possible to predict cognitive attributes like saliency and memorability from just an image or video, at the same time that advances in human-computer interaction and human cognition have made collecting human data more accessible than ever. In this work, we aim to reinforce the interplay between human perception and computational models. We develop new strategies for collecting perceptual data, build models that predict human responses to visual stimuli, and show how applications of these models can be used to prioritize content for human consumption. First, we develop a toolbox of web-based user interfaces for crowdsourcing attention data using only a laptop or mobile phone. Through experimentation and analysis, we show how to deploy these interfaces to collect attention data scalably and flexibly for a variety of use cases. Next, we use our toolbox to study a novel aspect of human attention, resulting in the first saliency model that is capable of producing multiple saliency heatmaps corresponding to different potential viewing durations. Finally, we turn our focus to memorability, by designing an online memory game to measure and predict how likely a person is to remember a video. Systems like these that are capable of modeling human perception can make intelligent decisions about what information to prioritize, create, enhance, and preserve.