2026-09-19T16:21:56Zhttps://dspace.mit.edu/server/oai/requestoai:dspace.mit.edu:1721.1/1572552024-10-10T04:00:05Zcom_1721.1_7582com_1721.1_7581col_1721.1_131023
Agrawal, Pulkit
Abhangi, Nishant
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
2024-10-09T18:31:30Z
2024-10-09T18:31:30Z
2024-09
2024-10-07T14:34:37.574Z
https://hdl.handle.net/1721.1/157255
Natural language instruction following capabilities are important for robots to follow tasks specified by human commands. Hence, many language conditioned robots have been trained on a wide variety of datasets with tasks annotated by natural language instructions. However, these datasets are often limited in their size and hence the distribution and nature of the instructions given by real world users might be different from that in the datasets. This makes it unclear how these robots will perform in real world environments. Hence, a large scale evaluation with diverse instructions is needed to benchmark the performance of these robots. However, using humans to collect more annotations is prohibitively expensive. We show that recent large language models provide a scalable and inexpensive way to do such an evaluation. Moreover, there is a large performance drop in robots when evaluated on this larger set of instructions. We also show that we can use different prompts to LLMs to control properties such as diversity of the generated instructions.
M.Eng.
Massachusetts Institute of Technology
In Copyright - Educational Use Permitted
Copyright retained by author(s)
https://rightsstatements.org/page/InC-EDU/1.0/
Red Teaming Language Conditioned Robotic Behavior
Thesis
application/pdf
Master
Master of Engineering in Electrical Engineering and Computer Science
Publication
unknown
unknown
<Publication xmlns="https://www.openaire.eu/cerif-profile/1.1/" id="287734f2-35ba-4d9c-9260-ed4ecfbcc37c">
<Type xmlns="https://www.openaire.eu/cerif-profile/vocab/COAR_Publication_Types">http://purl.org/coar/resource_type/c_1843</Type>
<Title>Red Teaming Language Conditioned Robotic Behavior</Title>
<PublishedIn>
<Publication>
</Publication>
</PublishedIn>
<PublicationDate>2024-09</PublicationDate>
<Authors>
<Author>
<DisplayName>Abhangi, Nishant</DisplayName>
<Affiliation>
<OrgUnit>
</OrgUnit>
</Affiliation>
</Author>
</Authors>
<Editors>
</Editors>
<Publishers>
<Publisher>
<DisplayName>Massachusetts Institute of Technology</DisplayName>
<OrgUnit />
</Publisher>
</Publishers>
<License>https://rightsstatements.org/page/InC-EDU/1.0/</License>
<Abstract>Natural language instruction following capabilities are important for robots to follow tasks specified by human commands. Hence, many language conditioned robots have been trained on a wide variety of datasets with tasks annotated by natural language instructions. However, these datasets are often limited in their size and hence the distribution and nature of the instructions given by real world users might be different from that in the datasets. This makes it unclear how these robots will perform in real world environments. Hence, a large scale evaluation with diverse instructions is needed to benchmark the performance of these robots. However, using humans to collect more annotations is prohibitively expensive. We show that recent large language models provide a scalable and inexpensive way to do such an evaluation. Moreover, there is a large performance drop in robots when evaluated on this larger set of instructions. We also show that we can use different prompts to LLMs to control properties such as diversity of the generated instructions.</Abstract>
<Access xmlns="http://purl.org/coar/access_right"
>
</Access>
</Publication>