VirtualHome: Building Socially Intelligent Agents via Simulation

Abstract

While remarkable progress has been made in building autonomous agents that can help us at complex tasks, these have typically been studied in isolated environments. To build agents that can be deployed in the real world, we need to study them in more human-centric settings, where they can interact with other humans and agents. Moreover, for agents to assist us effectively, they need to be able to operate in these environments while understanding our intentions and beliefs, and learn to coordinate with us effectively and safely.

This thesis investigates the development of such assistive agents through simulation environments. In the first part of the thesis, we introduce VirtualHome, a multi-agent platform for simulating human activities in household environments, and introduce a knowledge base of daily human activities that can be executed in the simulator. Then, we present agents that can perform different tasks in the environment given human descriptions or demonstrations of the activity. Finally, we study agents that can perform activities together with other humans in the environment. We propose a framework to simulate humans in the environment at scale and propose two challenges, Watch-And-Help and Online-Watch-And-Help, to benchmark the performance of different assistive agents, and test their effectiveness in assisting real humans performing activities in VirtualHome. Together, the methods and tools presented in this thesis provide a way to study assistive agents in simulation, allowing to develop these agents safely and at scale before deploying them into the real world.