Predictive vision

Abstract

Anticipating outcomes is the root of intelligence. This thesis investigates Predictive Vision with the goal to develop robust methods that anticipate the next events that may happen in images or videos. Importantly, we develop methods for eciently scaling learning algorithms to learn an extensive set of rules that enable richer visual understanding. While large annotated datasets fuel progress in object recognition, the knowledge required for event understanding is vast and potentially ambiguous. To tackle this challenge, we develop predictive vision algorithms that instead learn these rules directly from large amounts of raw, unlabeled data. Capitalizing on millions of natural videos, this work develops algorithms that learn to anticipate the visual future, forecast human actions, and recognize ambient sounds.