Non-Invasive Vision-Based Measurement of Hand Kinematics and Interaction

Abstract

The ability to manipulate and interact with the world is a key part of what distinguishes humanity from other animals, and the human hand is perhaps the most powerful tool we have to do so. As such, a great deal of research goes towards better understanding how the hand behaves during interaction tasks. Study of physical interaction requires measurement of hand kinematics and interaction forces. Unfortunately, current methods involve cumbersome sensors or external forces that inherently change the way that the subject behaves. In order to avoid these confounding factors, this thesis presents an approach to measuring hand kinematics, dynamics, and physical interaction using a non-encumbering vision-based tool.

The proposed tool consists of (1) Vision-based hand tracking of hand kinematics in joint space (2) Synergy extraction and synergy space projection (3) Visual soft tissue deformation based contact detection (4) An exploration of force estimation at the fingertips.

This pipeline is applied to a piano-based experiment for validation and comparison with existing tools. The results indicate that vision-based kinematics measurement is largely comparable to and at times shows more sensitivity to joint angle variation than traditionally instrumented approaches. However, force estimation is not yet a consistent alternative to physical sensor interfaces.