Shells and Stages for Actuated TUIs: Reconfiguring and Orchestrating Dynamic Physical Interaction

Abstract

Research on Actuated and Shape-Changing Tangible User Interfaces (TUIs) in the field of Human Computer Interaction (HCI) has been explored widely to design embodied interactions using digital computation has been explored widely. While advanced technical approaches, such as robotics and material science, have led to many concrete instances of Actuated TUIs, a single actuated hardware system, in reality, is inherently limited by its fixed configuration, thus limiting the reconfigurability, adaptability, and expressibility of its interactions.

In my thesis, I introduce novel hardware augmentation methods, Shells and Stages, for Actuated TUI hardware to expand and enrich their interactivity and expressibility for dynamic physical interactions. Shells act as passive mechanical attachments for Actuated TUIs that can extend, reconfigure and augment the interactivity and functionality of the hardware. Stages are physical platforms that allow Actuated TUIs to propel on a platform to create novel physical expression based on the duality of front stage and back stage. These approaches are inspired by theatrical performances, computational and robotic architecture, biological systems, physical tools and science fiction. While Shells and Stages can individually augment the interactivity and expressibility of the Actuated TUI system, the combination of the two enhances advanced physical expression based on combined shell-swaping and stage-transitioning. By introducing these novel modalities of Shells and Stages, the thesis expands and contributes to a new paradigm of Inter-Material / Device Interaction in the domain of Actuated TUIs.

The thesis demonstrates the concepts of Shells and Stages based on existing Actuated TUI hardware, including pin-based shape displays and self-propelled swarm user interfaces. Design and implementation methods are introduced to fabricate mechanical shells with different properties, and to orchestrate a swarm of robots on the stage with arbitrary configurations. To demonstrate the expanded interactivity and reconfigurability, a variety of interactive applications are presented via prototypes, ranging from digital data interaction, reconfigurable physical environment, storytelling, and tangible gaming. Overall, my research introduces a new A-TUI design paradigm that incorporates the self-actuating hardware (Actuated TUIs) and passively actuated mechanical modules (Shells) together with surrounding physical platforms (Stages). By doing so, my research envisions the future in which computational technology is coupled seamlessly with our physical environment. This next generation of TUIs, by interweaving multiple HCI research streams, aims to provide endless possibilities for reconfigurable tangible and embodied interactions enabled by fully expressive and functional movements and forms.