Contact-aware and multi-modal robotic manipulation

Abstract

Intelligent robotic manipulation has advanced significantly in recent years, driven by progress in foundational cognitive models, sensor-fusion techniques, and improvements in actuators and sensors. However, most contemporary robotic systems still lack the ability to effectively recognize and understand contact dynamics, which are critical for performing manipulation tasks beyond basic pick-and-place operations. This thesis argues and proves that contact awareness is essential for the successful deployment of robotic systems, not only in structured environments such as factories but also in unstructured settings like domestic households. Achieving contact awareness necessitates advancements in three key areas: the development of improved contact-sensing hardware, the creation of more expressive frameworks for representing and interpreting contact information, and the design of efficient modality-fusion algorithms to integrate these capabilities into robotic action planning. This work addresses these challenges by (1) proposing novel mechanical designs that enable touch sensors to adopt more compact and versatile forms while enhancing their durability and manufacturability, (2) introducing a foundational representation learning framework capable of learning a shared tactile latent representation, which can be transferred across different sensors and downstream tasks, and (3) developing a compositional diffusion-based approach for action prediction that integrates tactile sensing signals with other perception modalities, thereby enabling learning across diverse environments and promoting policy reuse. Along the way, this thesis demonstrates that tactile sensors can be both compact and versatile, challenging common perceptions to the contrary. It also establishes that tactile sensing is indispensable not only for high-precision tasks, such as electronics assembly, but also for everyday activities, including cooking and tool usage.