The SUN Action database : collecting and analyzing typical actions for visual scene types

Abstract

Recent work in human and machine vision has increasingly focused on the problem of scene recognition. Scene types are largely defined by the actions one might typically do there: an office is a place someone would typically "work". I introduce the SUN Action database (short for "Scene UNderstanding - Action"): the first effort to collect and analyze free-response data from human subjects about the typical actions associated with different scene types. Responses were gathered on Mechanical Turk for twenty images per catgory, each depicting a characteristic view of one of 397 different scene types. The distribution of phrases is shown to be heavy-tailed and Zipf-like, whereas the distribution of semantic roots is not Zipf-like. Categories strongly associated with particular tasks or actions are shown to have lower overall diversity of responses. A hierarchical clustering analysis reveals a heterogeneous clustering structure, with some categories readily grouping together, and other categories remaining apart even at coarse clustering levels. Finally, two simple classifiers are introduced for predicting scene types from associated actions: a nearest centroid classifier, and an empirical maximum likelihood classifier. Both classifiers demonstrate greater than 50% classification performance in a 397-way classification task.