Interaction Configurations and Prompt Guidance in Conversational AI for Question Answering in Human-AI Teams
Author(s)
Song, Jaeyoon
Ashktorab, Zahra
Pan, Qian
Dugan, Casey
Geyer, Werner
Malone, Thomas
Date Issued
October 16, 2025
Journal
Proceedings of the ACM on Human-Computer Interaction
Publisher
ACM
Citation
Jaeyoon Song, Zahra Ashktorab, Qian Pan, Casey Dugan, Werner Geyer, and Thomas W. Malone. 2025. Interaction Configurations and Prompt Guidance in Conversational AI for Question Answering in Human-AI Teams. Proc. ACM Hum.-Comput. Interact. 9, 7, Article CSCW305 (November 2025), 27 pages.
Version
Final published version
Abstract
Understanding the dynamics of human-AI interaction in question answering is crucial for enhancing collaborative efficiency. Extending from our initial formative study, which revealed challenges in human utilization of conversational AI support, we designed two configurations for prompt guidance: a Nudging approach, where the AI suggests potential responses for human agents, and a Highlight strategy, emphasizing crucial parts of reference documents to aid human responses. Through two controlled experiments, the first involving 31 participants and the second involving 106 participants, we compared these configurations against traditional human-only approaches, both with and without AI assistance. Our findings suggest that effective human-AI collaboration can enhance response quality, though merely combining human and AI efforts does not ensure improved outcomes. In particular, the Nudging configuration was shown to help improve the quality of the output when compared to AI alone. This paper delves into the development of these prompt guidance paradigms, offering insights for refining human-AI collaborations in conversational question-answering contexts and contributing to a broader understanding of human perceptions and expectations in AI partnerships.
MIT Department
Sloan School of Management
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3757486
