Compositional Robot Learning for Generalizable Interactions

Abstract

To understand environments effectively and to interact safely with humans, robots must generalize their learned models to scenarios they have never been trained on before, such as new commands and new agents. Humans have shown a remarkable ability to compose concepts they have learned before in order to interpret and to act in a novel environment. In contrast, many deep-learning based methods fail at compositional generalization, i.e., an ability to generalize to novel combinations of concepts that have not been seen before in training. This thesis presents several learning-based approaches that leverage compositionally to enable generalization in various reasoning skills such as language understanding and social interactions. First, we show how we can derive networks directly from compositional linguistic structures to enable robots to follow novel commands and act rationally in new scenarios. These networks are extended to encode temporal relationships in linear temporal logic and augmented with sampling-based planners to plan in continuous space. Then we show how we can formulate social interactions as reward operations and apply recursive reward estimation to enable robots to reason about novel social interactions. Finally, we explore an alternative approach to incorporate compositionality when there is no explicit compositional structure — using language as an intermediate representation for internal reasoning. We show how this linguistic representation can help robots predict the trajectories of other agents.