Show simple item record

dc.contributor.authorKuo, Yen-Ling
dc.contributor.authorKatz, Boris
dc.contributor.authorBarbu, Andrei
dc.date.accessioned2022-03-24T16:53:23Z
dc.date.available2022-03-24T16:53:23Z
dc.date.issued2020-05-31
dc.identifier.urihttps://hdl.handle.net/1721.1/141354
dc.description.abstractWe demonstrate how a sampling-based robotic planner can be augmented to learn to understand a sequence of natural language commands in a continuous configuration space to move and manipu- late objects. Our approach combines a deep network structured according to the parse of a complex command that includes objects, verbs, spatial relations, and attributes, with a sampling-based planner, RRT. A recurrent hierarchical deep network controls how the planner explores the environment, de- termines when a planned path is likely to achieve a goal, and estimates the confidence of each move to trade off exploitation and exploration between the network and the planner. Planners are designed to have near-optimal behavior when information about the task is missing, while networks learn to ex- ploit observations which are available from the environment, making the two naturally complementary. Combining the two enables generalization to new maps, new kinds of obstacles, and more complex sentences that do not occur in the training set. Little data is required to train the model despite it jointly acquiring a CNN that extracts features from the environment as it learns the meanings of words. The model provides a level of interpretability through the use of attention maps allowing users to see its reasoning steps despite being an end-to-end model. This end-to-end model allows robots to learn to follow natural language commands in challenging continuous environments.en_US
dc.description.sponsorshipThis material is based upon work supported by the Center for Brains,Minds and Machines (CBMM), funded by NSF STC award CCF-1231216.en_US
dc.publisherCenter for Brains, Minds and Machines (CBMM), Computation and Systems Neuroscience (Cosyne)en_US
dc.relation.ispartofseriesCBMM Memo;124
dc.titleDeep compositional robotic planners that follow natural language commandsen_US
dc.typeArticleen_US
dc.typeTechnical Reporten_US
dc.typeWorking Paperen_US


Files in this item

Thumbnail

This item appears in the following Collection(s)

Show simple item record