A uniform representation for visual concepts

Abstract

We present a method for learning visually-grounded word meanings, given as input a set of videos paired with natural-language sentences describing them. Our method uses a uniform feature representation for all words and word types rather than relying on handcrafted features specific to each word. We learn words in a weakly-supervised manner, with no need for annotated bounding boxes around objects of interest. We encode words as Hidden Markov models such that word models can be composed according to a sentence's semantic structure to efficiently recognize events in videos. We use a discriminative variant of Baum-Welch to learn the parameters for our word models, and demonstrate that our approach is able to learn words capturing appearance, spatial relations, and temporal dynamics.