NDF-Based API for Human-assisted Language Planning (HaLP)
Name
fong-afo-meng-eecs-2023-thesis.pdf
Description
Thesis PDF
Size
15.47 MB
Format
Adobe PDF
Checksum (MD5)
b9148a9c620cfc7a806c04420df152e4
Author(s)
Fong, Alisha
Advisor(s)
Agrawal, Pulkit
Date Issued
June 2023
Publisher
Massachusetts Institute of Technology
Abstract
Recent works have show the promise of LLMs for generalizable task planning. Challenges in integrating LLM for high-level planning include outputting infeasible or sub-optimal plans, but the potentials include cultural commonsense to reason about high-level tasks on par with a human. Related works will generate free-form text that may contain actions inaccessible to the robot or over constrain the planner by providing it a static set of possible actions to select from and output mediocre plans. Humans can also decide when a task is infeasible due to limitations in the action space, and try to propose alternative plans. We show that LMs are also able to. We present an LLM-planner with the ability to request online learning of skills to output and execute optimal tabletop manipulation plans, even when the initial set of robot skills is insufficient. We build a fullstack system and deploy our method in simulation and hardware to demonstrate the capabilities of the planner and the preference for these plans over others in our ablation experiments. To support the learning of new skills, we present a low-level control API conditioned on natural language using Neural Descriptor Fields (NDFs) for out-of-plane category level manipulation that is SE(3)-equivariant and highly data-efficient to enable online learning.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
