Deformable Object Manipulation with a Tactile Reactive Gripper

Abstract

Manipulating deformable objects remains a fundamental challenge in robotics, as techniques developed for rigid objects often fail to generalize. Deformable objects exhibit infinite-dimensional configuration spaces, frequent self-occlusion, and high model uncertainty, making global state estimation and predictive modeling unreliable. To address these challenges, we propose a perception-driven framework that combines global visual understanding with local tactile feedback. Rather than modeling the full configuration of the object, we leverage local constraints, grounded in modular visual and tactile representations, to enable robust, reactive, and generalizable manipulation. The primary contributions of this work include: • Chapter 2: Cable Following. A tactile control strategy for in-hand cable manipulation that decouples contact regulation from object pose control, enabling fast, reactive sliding and closed-loop plug insertion using only local tactile feedback. • Chapter 3: Towel Edge Tracing. An extension of contact-based control to fabric edge following and the learned tactile perception networks to support this capability. • Chapter 4: Visuotactile Grasp Affordance. A grasp affordance model trained in simulation and refined with tactile self-supervision, enabling high-confidence edge grasping on towels. • Chapter 5: Dense Object Correspondence. A confidence-aware dense descriptor representation. Supports correspondence across crumpled and symmetric garments in air and on a table. • Chapter 6: Behavior Architecture and Planning Interfaces. Integration of perception modules into a reactive, confidence-based folding system and an exploration of how dense descriptors can interface with demonstrations, language, and task and motion planning. Collectively, these contributions show that global state estimation and dynamics prediction are not required for reliable deformable manipulation. Instead, semantically meaningful local interactions, guided by modular visual and tactile representations, can drive scalable, long-horizon behaviors across varied objects, configurations, and tasks.