Learning Semantic Maps from Natural Language Descriptions
Author(s)Walter, Matthew R.; Hemachandra, Sachithra Madhaw; Homberg, Bianca S.; Tellex, Stefanie; Teller, Seth
DownloadWalter_A framework.pdf (3.564Mb)
MetadataShow full item record
This paper proposes an algorithm that enables robots to efficiently learn human-centric models of their environment from natural language descriptions. Typical semantic mapping approaches augment metric maps with higher-level properties of the robot’s surroundings (e.g., place type, object locations), but do not use this information to improve the metric map. The novelty of our algorithm lies in fusing high-level knowledge, conveyed by speech, with metric information from the robot’s low-level sensor streams. Our method jointly estimates a hybrid metric, topological, and semantic representation of the environment. This semantic graph provides a common framework in which we integrate concepts from natural language descriptions (e.g., labels and spatial relations) with metric observations from low-level sensors. Our algorithm efficiently maintains a factored distribution over semantic graphs based upon the stream of natural language and low-level sensor information. We evaluate the algorithm’s performance and demonstrate that the incorporation of information from natural language increases the metric, topological and semantic accuracy of the recovered environment model.
DepartmentMassachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory; Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Proceedings of the 2013 Robotics: Science and Systems IX Conference
Robotics: Science and Systems
Walter, Matthew R., Sachithra Hemachandra, Bianca Homberg, Stefanie Tellex, and Seth Teller. "Learning Semantic Maps from Natural Language Descriptions." Proceedings of the 2013 Robotics: Science and Systems IX Conference, June 24-28, 2013, Berlin, Germany.
Final published version