LLMR: Real-time Prompting of Interactive Worlds using Large Language Models
Name
3613904.3642579.pdf
Size
17.08 MB
Format
Adobe PDF
Checksum (MD5)
623ca30c072324424800b6fba82b3a84
Author(s)
De La Torre, Fernanda
Fang, Cathy Mengying
Huang, Han
Banburski-Fahey, Andrzej
Amores Fernandez, Judith
Lanier, Jaron
Date Issued
May 11, 2024
Publisher
ACM
Citation
De La Torre, Fernanda, Fang, Cathy Mengying, Huang, Han, Banburski-Fahey, Andrzej, Amores Fernandez, Judith et al. 2024. "LLMR: Real-time Prompting of Interactive Worlds using Large Language Models."
Version
Final published version
Abstract
We present Large Language Model for Mixed Reality (LLMR), a framework for the real-time creation and modification of interactive Mixed Reality experiences using LLMs. LLMR leverages novel strategies to tackle difficult cases where ideal training data is scarce, or where the design goal requires the synthesis of internal dynamics, intuitive analysis, or advanced interactivity. Our framework relies on text interaction and the Unity game engine. By incorporating techniques for scene understanding, task planning, self-debugging, and memory management, LLMR outperforms the standard GPT-4 by 4x in average error rate. We demonstrate LLMR’s cross-platform interoperability with several example worlds, and evaluate it on a variety of creation and modification tasks to show that it can produce and edit diverse objects, tools, and scenes. Finally, we conducted a usability study (N=11) with a diverse set that revealed participants had positive experiences with the system and would use it again.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3613904.3642579
