NDF-Based API for Human-assisted Language Planning (HaLP)

Abstract

Recent works have show the promise of LLMs for generalizable task planning. Challenges in integrating LLM for high-level planning include outputting infeasible or sub-optimal plans, but the potentials include cultural commonsense to reason about high-level tasks on par with a human. Related works will generate free-form text that may contain actions inaccessible to the robot or over constrain the planner by providing it a static set of possible actions to select from and output mediocre plans. Humans can also decide when a task is infeasible due to limitations in the action space, and try to propose alternative plans. We show that LMs are also able to. We present an LLM-planner with the ability to request online learning of skills to output and execute optimal tabletop manipulation plans, even when the initial set of robot skills is insufficient. We build a fullstack system and deploy our method in simulation and hardware to demonstrate the capabilities of the planner and the preference for these plans over others in our ablation experiments. To support the learning of new skills, we present a low-level control API conditioned on natural language using Neural Descriptor Fields (NDFs) for out-of-plane category level manipulation that is SE(3)-equivariant and highly data-efficient to enable online learning.