Realtime state estimation for contact manipulation

Abstract

In this thesis, we develop a real-time state estimation system to recover the pose and contact state of an object relative to its environment. The capability to make such estimations is important for a controller to adequately react to uncertainties in a manipulation task. We propose a framework that combines tactile and visual sensing to improve the accuracy and robustness. Visual sensing is versatile and non-intrusive but suffers from occlusions and limited accuracy, especially with regard to tasks involving contact. Tactile sensing (including contact and force) is local but provides accuracy and robustness to occlusions. The framework uses online estimation techniques to fuse kinematic measurements made by a robot, contact geometry of the object and the environment, and visual measurements. In a complex contact task such as insertion, the contact formations are hard to resolve directly. We propose a data-driven method to assess the contact formation, which is then used in real time by the state estimator. We apply our framework to two iconic tasks in robotic manipulation: planar pushing and object insertion. We evaluate the algorithm in a setup instrumented to provide ground truth. The experiments show that our approach provides an accurate and robust estimation for the studied manipulation tasks.