2026-09-19T06:15:36Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1394162022-01-15T03:57:19Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Shah, Julie A.
Figueroa, Nadia
Kim, Dain
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2022-01-14T15:10:13Z
2022-01-14T15:10:13Z
2021-06
2021-06-17T20:13:32.193Z
https://hdl.handle.net/1721.1/139416
As robots are increasingly being utilized to perform automated tasks, effective methods for transferring task specifications to robots have become imperative. However, existing techniques for training robots to perform tasks often depend on rote mimicry of human demonstrations and do not generalize well to new tasks or contexts. In addition, learning an end-to-end policy for performing a sequence of operations for a high-level goal remains a challenge. Transferring sequential task specifications is a difficult objective, as it requires extensive human intervention to establish the structure of the task including the constraints, objects of interest, and control parameters.
In this thesis, we present an imitation learning framework for sequential manipulation tasks that enables humans to easily communicate abstract high-level task goals to the robot without explicit programming or robotics expertise. We introduce natural language input to the system to facilitate the learning of task specifications. During training, a human teacher provides demonstrations and a verbal description of the task being performed. The training process then learns a mapping from the multi-modal inputs to the low-level control policies. During execution, the high-level task instruction input is parsed into a list of sub-tasks that the robot has learned to perform.
The presented framework is evaluated in a simulated table-top scenario of a robotic arm performing sorting and kitting tasks from natural language commands. The approach developed in this thesis achieved an overall task completion rate of 91.16% on 600 novel task scenes, with a sub-task execution success rate of 96.44% on 1,712 individual “pick” and “place” tasks.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright MIT
http://rightsstatements.org/page/InC-EDU/1.0/
Imitation Learning for Sequential Manipulation Tasks: Leveraging Language and Perception
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="34ab2cf0-9a16-43e0-bc54-db88d0283a1d">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Imitation Learning for Sequential Manipulation Tasks: Leveraging Language and Perception</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2021-06</PublicationDate>
<Authors>
<Author>
<DisplayName>Kim, Dain</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>http://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>As robots are increasingly being utilized to perform automated tasks, effective methods for transferring task specifications to robots have become imperative. However, existing techniques for training robots to perform tasks often depend on rote mimicry of human demonstrations and do not generalize well to new tasks or contexts. In addition, learning an end-to-end policy for performing a sequence of operations for a high-level goal remains a challenge. Transferring sequential task specifications is a difficult objective, as it requires extensive human intervention to establish the structure of the task including the constraints, objects of interest, and control parameters.
In this thesis, we present an imitation learning framework for sequential manipulation tasks that enables humans to easily communicate abstract high-level task goals to the robot without explicit programming or robotics expertise. We introduce natural language input to the system to facilitate the learning of task specifications. During training, a human teacher provides demonstrations and a verbal description of the task being performed. The training process then learns a mapping from the multi-modal inputs to the low-level control policies. During execution, the high-level task instruction input is parsed into a list of sub-tasks that the robot has learned to perform.
The presented framework is evaluated in a simulated table-top scenario of a robotic arm performing sorting and kitting tasks from natural language commands. The approach developed in this thesis achieved an overall task completion rate of 91.16% on 600 novel task scenes, with a sub-task execution success rate of 96.44% on 1,712 individual “pick” and “place” tasks.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>