Robust Dexterous Manipulation Enabled by Learning at Scale inSimulation

Abstract

Robots with robust bimanual dexterity have the potential to transform industries such as manufacturing and healthcare by performing complex tasks at human-level proficiency. While end-to-end learning methods have shown promise in achieving this goal, scaling these approaches remains challenging. Existing paradigms suffer from high costs associated with collecting large-scale, high-quality demonstrations on physical systems and face performance saturation due to reliance on offline data. We propose a task-agnostic pipeline that leverages robotics simulation to overcome these limitations. In particular, we introduce DART, a cost-effective, augmented reality, robot teleoperation platform for scalable data collection. We demonstrate through user study that it enables twice the throughput of existing systems. We also present a learning algorithm that integrates real-world demonstrations with reinforcement learning to surpass performance plateaus. Finally, we design a method that zero-shot transfers policies trained in simulation on real robots using only RGB input. Together, these contributions provide a practical and scalable path toward achieving general-purpose dexterous robot manipulation.