The Discovery of perceptual structure from visual co-occurrences in space and time

Abstract

Although impressionists assure us that the world is just dabs of light, we cannot help but see surfaces and contours, objects and events. How can a visual system learn to organize pixels into these higher-level structures? In this thesis I argue that perceptual organization reflects statistical regularities in the environment. When visual primitives occur together much more often than one would expect by chance, we may learn to associate those primitives and to form a perceptual group. The first half of the thesis deals with the identification of such groups at the pixel level. I show that low-level image statistics are surprisingly effective at higher-level segmentation. I present an algorithm that groups pixels by identifying meaningful co-occurrences in an image's color statistics. Consider a zebra. Black-next-to-white occurs suspiciously often, hinting that these colors have a common cause. I model these co-occurrences using pointwise mutual information (PMI). If the PMI between two colors is high, then the colors probably belong to the same object. Grouping pixels with high PMI reveals object segments. Separating pixels with low PMI marks perceived boundaries. If simple color co-occurrences can tell us about object segments, what might more complex statistics tell us? The second half of the thesis investigates high dimensional visual data, such as image patches and video frames. In high dimensions, it is intractable to directly model co-occurrences. Instead, I show that modeling PMI can be posed as a simpler binary classification problem in which the goal is to predict if two primitives occur in the same spatial or temporal context. This allows us to model PMI associations between complex inputs. I demonstrate the effectiveness of this approach on three domains: discovering objects by associating image patches, discovering movie scenes by associating frames, and discovering place categories by associating geotagged photos. Together, these results shed light on how a visual system can learn to organize raw sensory input into meaningful percepts.