VirtualHome : learning to infer programs from synthetic videos of activities in the home

Abstract

This project models complex activities that occur in a typical household. Programs - sequences of atomic actions and interactions - are used as a high-level, unambiguous representation of complex activities executable by an agent. However, no dataset of household activity programs currently exists. This project builds such a dataset by crowdsourcing programs of typical household activities, via a game-like interface used for teaching kids how to code. The most common atomic actions are implemented in the Unity3D game engine, and videos are recorded of an agent executing the collected programs in a simulated household environment. The VirtualHome simulator allows the creation of a large activity video dataset with rich groundtruth, enabling training and testing of video understanding models. Using the collected dataset, a sequence-to-sequence neural encoder-decoder model with visual attention is built, and learns to infer programs directly from videos. It achieves 46.2% accuracy for action inference.