Object schemas for grounding language in a responsive robot
Author(s)Hsiao, Kai-yuh; Tellex, Stefanie; Vosoughi, Soroush; Kubat, Rony Daniel; Roy, Deb K.
MetadataShow full item record
An approach is introduced for physically grounded natural language interpretation by robots that reacts appropriately to unanticipated physical changes in the environment and dynamically assimilates new information pertinent to ongoing tasks. At the core of the approach is a model of object schemas that enables a robot to encode beliefs about physical objects in its environment using collections of coupled processes responsible for sensorimotor interaction. These interaction processes run concurrently in order to ensure responsiveness to the environment, while co-ordinating sensorimotor expectations, action planning and language use. The model has been implemented on a robot that manipulates objects on a tabletop in response to verbal input. The implementation responds to verbal requests such as ‘Group the green block and the red apple’, while adapting in real time to unexpected physical collisions and taking opportunistic advantage of any new information it may receive through perceptual and linguistic channels.
DepartmentMassachusetts Institute of Technology. Media Laboratory; Program in Media Arts and Sciences (Massachusetts Institute of Technology)
Hsiao, Kai-yuh, Stefanie Tellex, Soroush Vosoughi, Rony Kubat, and Deb Roy. “Object Schemas for Grounding Language in a Responsive Robot.” Connection Science 20, no. 4 (December 2008): 253–276.
Author's final manuscript