Automatic, Careful Online Packing of Groceries Using a Soft Robotic Manipulator and Multimodal Sensing

Abstract

This thesis describes the use of soft robotic manipulators with multimodal sensing for estimating the physical properties of unknown objects to enable sorting and packing. Although bin packing has been a key benchmark task for robotic manipulation, the community has mainly focused on the placement of rigid rectilinear objects within the container. We address this by presenting a soft robotic hand that uses a combination of vision, motor-based proprioception and soft tactile sensors to identify and pack a stream of unknown objects. We translate the ill-defined human conception of a “well-packed container” into metrics that match combinations of our different sensor modalities and demonstrate how this works in a grocery packing scenario, where objects of arbitrary shape, size and stiffness come down a conveyor belt. The proposed multimodal approach is supported by physical experiments demonstrating how the integration of multiple sensing modalities can address complex manipulation applications.