Red Teaming Language Conditioned Robotic Behavior
Name
abhangi-abhangin-meng-eecs-2024-thesis.pdf
Description
Thesis PDF
Size
654.67 KB
Format
Adobe PDF
Checksum (MD5)
bcbde898cf1fcf1ad2a2dd1423f06558
Author(s)
Abhangi, Nishant
Advisor(s)
Agrawal, Pulkit
Date Issued
September 2024
Publisher
Massachusetts Institute of Technology
Abstract
Natural language instruction following capabilities are important for robots to follow tasks specified by human commands. Hence, many language conditioned robots have been trained on a wide variety of datasets with tasks annotated by natural language instructions. However, these datasets are often limited in their size and hence the distribution and nature of the instructions given by real world users might be different from that in the datasets. This makes it unclear how these robots will perform in real world environments. Hence, a large scale evaluation with diverse instructions is needed to benchmark the performance of these robots. However, using humans to collect more annotations is prohibitively expensive. We show that recent large language models provide a scalable and inexpensive way to do such an evaluation. Moreover, there is a large performance drop in robots when evaluated on this larger set of instructions. We also show that we can use different prompts to LLMs to control properties such as diversity of the generated instructions.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
