Talk to the Hand: an LLM-powered Chatbot with Visual Pointer as Proactive Companion for On-Screen Tasks
Name
3706598.3715579.pdf
Size
2.89 MB
Format
Adobe PDF
Checksum (MD5)
9b17b8610c3e7c7c3192f2c58ea673c1
Author(s)
Prasongpongchai, Thanawit
Pataranutaporn, Pat
Lertsutthiwong, Monchai
Maes, Pattie
Date Issued
April 25, 2025
Publisher
ACM|CHI Conference on Human Factors in Computing Systems
Citation
Thanawit Prasongpongchai, Pat Pataranutaporn, Monchai Lertsutthiwong, and Pattie Maes. 2025. Talk to the Hand: an LLM-powered Chatbot with Visual Pointer as Proactive Companion for On-Screen Tasks. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). Association for Computing Machinery, New York, NY, USA, Article 637, 1–16.
Version
Final published version
Abstract
This paper presents Pointer Assistant, a novel human-AI interaction technique for on-screen tasks. The design features a chatbot displayed as an extra mouse pointer, alongside the user’s, which proactively gives feedback on user actions while directing them to relevant areas on the screen and responding to the user’s direct chat messages. The effectiveness of the design’s key characteristics, pointer form and proactivity, was investigated in a study involving 220 participants in a financial budget planning task. Results demonstrated that the pointer design and interaction reduced task load while improving satisfaction with the experience, and increased the number of budget categories ideated during the task compared to the traditional passive chat log design. Participants viewed Pointer Assistant as a fun, innovative, and helpful visual guide while noting that its assertiveness can be improved. Future developments could offer even further enhancements to the user experience of human-AI collaboration and task outcomes.
Description
CHI ’25, Yokohama, Japan
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3706598.3715579
