Integrated perception, planning and feedback control for generalizable robotic manipulation

Abstract

Humans can easily adapt their manipulation skills to unseen objects, new environment and different tasks. However, existing robot manipulators are typically limited to known object instance and skill transferring is challenging. In this thesis, we take a step further by formulating a manipulation framework that can achieve precise, reliable and dexterous manipulation while being generalizable to potentially unknown object instances. To achieve it, we propose key-point affordances, an object representation consists of 3D semantic key-points. This object representation captures task-related geometric information while ignoring irrelevant details, which enables our method to handle unknown objects with potentially large shape variations. We implement perception, planning and feedback control modules on top of key-point affordances and integrate them into a fully functionally perception-to-action manipulation pipeline. Extensive experiments demonstrate our method can reliably accomplish a variety of challenging tasks with never-before seen objects.