Exploring controllable symbolic music generation for real-time
embodied interactions and performances
Name
lei-hlei-meng-eecs-2026-thesis.pdf
Size
2.79 MB
Format
Adobe PDF
Checksum (MD5)
90bc4dd3a2ab9f4819f396da9ddb4da3
Author(s)
Lei, Heidi
Advisor(s)
Huang, Cheng-Zhi Anna
Date Issued
February 2026
Publisher
Massachusetts Institute of Technology
Abstract
Transformer-based sequence models have enabled symbolic music generation with increasingly convincing structure and stylistic consistency, but their usefulness in creative practice depends critically on control. In real-time interactive settings such as improvisation and performance where humans and AI collaborate in their artistic creation, the desired musical direction can change continuously and latency constraints preclude expensive optimization at inference time. This thesis investigates inference-time steering for transformer-based symbolic music generation, focusing on methods that influence the model’s output distribution during decoding to support low-latency, musically meaningful control. We develop three controllers that target core musical dimensions: pitch contour control, tempo control, and harmonic control, through a real-time generation steering framework based on direct manipulation of token logits during autoregressive sampling as well as injecting new conditioning signal through a light-weight cross attention layer. We discuss the central challenge that arises in this setting: increasing steering strength improves responsiveness and constraint satisfaction, but can degrade coherence, stylistic plausibility, and continuity, particularly at boundaries where control signals change. To evaluate these trade-offs in realistic human-AI co-creative settings, we integrate the proposed controllers into two interactive systems. The first is a keyboard improvisation setup in which a performer steers generated material while maintaining a fluent improvisational workflow. The second is a live dance performance system in which music is steered through embodied movement signals, revealing both expressive opportunities and practical constraints associated with noisy sensing and mapping design. Across the case studies, we identify recurring patterns in system behavior and user experience, including the limits of intentional control, the importance of stable boundary behavior, and the ways performers learn to negotiate agency with a generative model. Supplementary audio and visual materials are available online at https://people.csail.mit.edu/hlei/real-time-music-control/.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
