Following recipes with a cooking robot

Abstract

In this thesis, we present BakeBot, a PR2 robot system that interprets natural language baking recipes into baking instructions which it follows to execute the recipe, from mise en place presentation of the ingredients through baking in a toaster oven. We developed parameterized motion primitives for baking. The motion primitives utilize the existing sensing and manipulation capabilities of the PR2 platform and also our new compliant control techniques to address environmental uncertainty. The system was first implemented as a static finite state machine, which was tested through 27 baking attempts, 16 of which successfully resulted in edible cookies. The system was then implemented as a dynamic state machine, in which the robot estimated the world state and planned sequences of motion primitives to follow the baking instructions inferred from the natural language recipe, which was tested thorough five baking attempts of two different recipes, all of which resulted in edible cookies.