The Shape of Music. Computational Specification of Hand Gestures in Piano Playing.

Abstract

Essential parts of human communication, expression, and productive action rely on gestures. Examining these gestures is central to design and to understanding creativity. However, techniques and skills manifested in human hand gestures are hard to capture in computational terms. In this research, I investigated how skilled hand gestures can be described computationally and apprehended visually as shapes. A case study was carried out on piano performance. I explored piano hand gestures in terms of technique and expressive intent, essential elements of the pianist’s playing style.

I designed and ran a controlled study to capture gesture variation in performances of the same music, with six proficient piano players participating in the recordings. I developed technical workflows and processes to record multimodal performance data, including video, audio, MIDI and 3D motion capture, and prepared the data for visualization and analysis. I compared two performers’ piano techniques from their executions of technical exercises and detected elements of expression in a pianist’s performances of Prelude in C Minor, BWV.999 by J.S. Bach.

Experts evaluated the performances from audio and video. From their comments, I extracted qualities of technique and expression essential in piano playing. Then, I determined features to measure these qualities in the motion and MIDI data. I explored the recorded data by calculating the features for different partitions of the music score. I developed feature visualizations displaying the pianists’ playing patterns. To exemplify how motion and music data could be decomposed into gestures, I presented an outline for a shape grammar for parsing musical performances.

I concluded that expressive patterns detected in the played music and hand motion could be combined to identify and study gestures according to stylistic elements of piano performance. Overall, the study points towards a perceptual approach to determining and analyzing skilled hand gestures in piano playing and beyond.