Situated mappings : augmented-reality clay and adaptive interfacing

Abstract

Our collectively developed methods of structuring information through graphical and natural user interfaces (GUI, NUI) largely emphasize visual and visuo-spatial representation over other types of sensory information. As our interfaces continue to develop we seem to select for fidelity of visual stimulation, while neglecting the behavioral aspects of physical materiality. In this thesis, I advocate for the use of expressive mediums of material engagement as part of the design of interaction within interfaces. I present an approach to interfacing using Computer-Mediated Material Interaction (CMMI), by augmenting the visual appearance and behavior of clay. This approach considers the subject's situated learning as the crucial factor in building coherent and immersive interfacing. Taking precedents from the domains of perceptual science and cognitive ecology, such as information-search, affordances and adaptive learning, I suggest that the sense of immersion and ease of learning in an interaction stems from how much it supports the subject's self-guided, iterative, discovery process. To illustrate this approach, I present a series of oil clay augmented-reality prototypes that encourage situated mappings to be discovered by the subject. I also discuss how interface design could further honor the adaptive principles latent in human-material-interactions.