Learning to act with objects, relations and physics

Abstract

Humans display an unrivalled degree of control over their environments. From a young age, humans represent the world in ways that allow them to not just make inferences about how the world works, but also to act and intervene on the world in order to accomplish their goals. Even children can pick up a new skill like “catapulting” from a single demonstration or just a few trials of experience, while it might take a machine agent several hundreds, thousands, or even millions of attempts to master such a skill. For those focused on better understanding these human capabilities, or for those wishing to build more flexible and efficient machines, the computational question is the same: how do people learn and generalize to new problems from just a handful of experiences?

This thesis presents physical problem solving as a window on the flexibility and efficiency of human and machine action. Across two tasks introduced and studied in this thesis, the Gluing Task and the Virtual Tools game, structured action spaces and mental simulation are crucial to explaining human behavior. These action spaces are both object-oriented and relational, and their representations can be learned with techniques such as deep reinforcement learning or program induction to enable better generalization to new problems. By combining structured action spaces with mental simulation, humans and machines can be efficient in the number of actions they require to solve problems and compositionally integrate information gained by trial-and-error experience with information gained by passive observation. Embodied real world experience can additionally affect how much humans rely on mental simulation in physical problem solving. Individuals born with limb differences, like having only one hand, spend significantly more time thinking and less time acting when faced with a physical puzzle, perhaps reflecting a higher cost of action learned from their everyday experience. Taken together, these results suggest that the flexibility of human physical problem solving stems from the mental simulations people employ when faced with new problems, while the efficiency of human search rests upon appropriately structured action spaces that can be rapidly transformed through minimal trial-and-error experience.