Temporal Grounding Graphs for Language Understanding with Accrued Visual-Linguistic Context
Name
43c6f5f1a10c79985af40c0e2af903615fd1.pdf
Size
13.91 MB
Format
Adobe PDF
Checksum (MD5)
5170a41d01701248ba3bc19a65bb4bcf
Author(s)
Paul, Rohan
Barbu, Andrei
Felshin, Sue
Katz, Boris
Roy, Nicholas
Date Issued
August 2017
Journal
Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence
Publisher
International Joint Conferences on Artificial Intelligence
Citation
Paul, Rohan, Andrei Barbu, Sue Felshin, Boris Katz, and Nicholas Roy. “Temporal Grounding Graphs for Language Understanding with Accrued Visual-Linguistic Context.” Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence (August 2017).
Version
Author's final manuscript
Abstract
A robot's ability to understand or ground natural language instructions is fundamentally tied to its knowledge about the surrounding world. We present an approach to grounding natural language utterances in the context of factual information gathered through natural-language interactions and past visual observations. A probabilistic model estimates, from a natural language utterance, the objects, relations, and actions that the utterance refers to, the objectives for future robotic actions it implies, and generates a plan to execute those actions while updating a state representation to include newly acquired knowledge from the visual-linguistic context. Grounding a command necessitates a representation for past observations and interactions; however, maintaining the full context consisting of all possible observed objects, attributes, spatial relations, actions, etc., over time is intractable. Instead, our model, Temporal Grounding Graphs, maintains a learned state representation for a belief over factual groundings, those derived from natural-language interactions, and lazily infers new groundings from visual observations using the context implied by the utterance. This work significantly expands the range of language that a robot can understand by incorporating factual knowledge and observations of its workspace in its inference about the meaning and grounding of natural-language utterances.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Aeronautics and Astronautics
Terms of Use
Creative Commons Attribution-Noncommercial-Share Alike
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.24963/IJCAI.2017/629
