Robust object pose estimation with point clouds from vision and touch

Abstract

We present a study of object pose estimation performed with hybrid visuo-tactile sensing in mind. We propose that a tactile sensor can be treated as a source of dense local geometric information, and hence consider it to be a point cloud source analogous to an RGB-D camera. We incorporate the tactile geometric information directly into a conventional point-cloud-based articulated object tracker based on signed-distance functions. This tracker runs at 12 Hz using an online depth reconstruction algorithm for the GelSight tactile sensor and a modified second-order update for the tracking algorithm. The tracker provides robust pose estimates of small objects throughout manipulation, even when the objects are occluded by the robot's end effector. To address limitations in this tracker, we additionally present a formulation of the underlying point-cloud correspondence problem as a mixed-integer convex program, which we efficiently solve to optimality with an off-the-shelf branch and bound solver. We show that reasoning about object pose estimation in this way allows natural extension to point-to-mesh correspondence, multiple object estimation, and outlier rejection without losing the ability to obtain a globally optimal solution. We probe the extent to which rich problem-specific formulations typically tackled with unreliable nonlinear optimization can be rigorously treated in a global optimization framework.