G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios
Author(s)
Wang, Zeyu
Shi, Yuanchun
Wang, Yuntao
Yao, Yuchen
Yan, Kun
Wang, Yuhan
Ji, Lei
Xu, Xuhai
Yu, Chun
Date Issued
May 13, 2024
Journal
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Publisher
Association for Computing Machinery
Citation
Wang, Zeyu, Shi, Yuanchun, Wang, Yuntao, Yao, Yuchen, Yan, Kun et al. 2024. "G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios." Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 8 (2).
Version
Final published version
Abstract
Modern information querying systems are progressively incorporating multimodal inputs like vision and audio. However, the integration of gaze --- a modality deeply linked to user intent and increasingly accessible via gaze-tracking wearables --- remains underexplored. This paper introduces a novel gaze-facilitated information querying paradigm, named G-VOILA, which synergizes users' gaze, visual field, and voice-based natural language queries to facilitate a more intuitive querying process. In a user-enactment study involving 21 participants in 3 daily scenarios (p = 21, scene = 3), we revealed the ambiguity in users' query language and a gaze-voice coordination pattern in users' natural query behaviors with G-VOILA. Based on the quantitative and qualitative findings, we developed a design framework for the G-VOILA paradigm, which effectively integrates the gaze data with the in-situ querying context. Then we implemented a G-VOILA proof-of-concept using cutting-edge deep learning techniques. A follow-up user study (p = 16, scene = 2) demonstrates its effectiveness by achieving both higher objective score and subjective score, compared to a baseline without gaze data. We further conducted interviews and provided insights for future gaze-facilitated information querying systems.
MIT Department
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution-Noncommercial
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3659623
