Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration
Name
3640543.3645198.pdf
Size
1.23 MB
Format
Adobe PDF
Checksum (MD5)
fde368235e975c336f4397d45b813752
Author(s)
Qian, Crystal
Wexler, James
Date Issued
March 18, 2024
Publisher
ACM|29th International Conference on Intelligent User Interfaces
Citation
Qian, Crystal and Wexler, James. 2024. "Take It, Leave It, or Fix It: Measuring Productivity and Trust in Human-AI Collaboration."
Version
Final published version
Abstract
Although recent developments in generative AI have greatly enhanced the capabilities of conversational agents such as Google’s Bard or OpenAI’s ChatGPT, it’s unclear whether the usage of these agents aids users across various contexts. To better understand how access to conversational AI affects productivity and trust, we conducted a mixed-methods, task-based user study, observing 76 software engineers (N=76) as they completed a programming exam with and without access to Bard. Effects on performance, efficiency, satisfaction, and trust vary depending on user expertise, question type (open-ended "solve" questions vs. definitive "search" questions), and measurement type (demonstrated vs. self-reported). Our findings include evidence of automation complacency, increased reliance on the AI over the course of the task, and increased performance for novices on “solve”-type questions when using the AI. We discuss common behaviors, design recommendations, and impact considerations to improve collaborations with conversational AI.
Description
IUI ’24, March 18–21, 2024, Greenville, SC, USA
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3640543.3645198
