LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
Author(s)
De La Torre, Fernanda; Fang, Cathy Mengying; Huang, Han; Banburski-Fahey, Andrzej; Amores Fernandez, Judith; Lanier, Jaron; ... Show more Show less
Download3613904.3642579.pdf (17.08Mb)
Publisher with Creative Commons License
Publisher with Creative Commons License
Creative Commons Attribution
Terms of use
Metadata
Show full item recordAbstract
We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR’s cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.
Date issued
2024-05-11Publisher
ACM
Citation
De La Torre, Fernanda, Fang, Cathy Mengying, Huang, Han, Banburski-Fahey, Andrzej, Amores Fernandez, Judith et al. 2024. "LLMR: Real-time Prompting of Interactive Worlds using Large Language Models."
Version: Final published version
ISBN
979-8-4007-0330-0
Collections
The following license files are associated with this item: