SenseMate: An Accessible and Beginner-Friendly Human-AI Platform for Qualitative Data Analysis
Name
3640543.3645194.pdf
Size
1.4 MB
Format
Adobe PDF
Checksum (MD5)
4395ad24c11efd35d194958fd1928f74
Author(s)
Overney, Cassandra
Saldías, Belén
Dimitrakopoulou, Dimitra
Roy, Deb
Date Issued
March 18, 2024
Publisher
ACM
Citation
Overney, Cassandra, Saldías, Belén, Dimitrakopoulou, Dimitra and Roy, Deb. 2024. "SenseMate: An Accessible and Beginner-Friendly Human-AI Platform for Qualitative Data Analysis."
Version
Final published version
Abstract
Community organizations face challenges in harnessing the power of qualitative data analysis, or sensemaking, to understand the diverse perspectives and needs brought up by their constituents. One of the most time-consuming and tedious parts of sensemaking is qualitative coding, or the process of identifying themes across a large and unstructured corpus of community input. A challenge in qualitative coding is attaining high intercoder reliability, especially between expert and beginner sensemakers. In this work, we present SenseMate, a novel human-AI system designed to help with qualitative coding. SenseMate leverages rationale extraction models, a new machine learning strategy to semi-automate sensemaking, which produces theme recommendations and human-interpretable explanations. The models were trained on a dataset of people’s experiences living in Boston, which was annotated for themes by expert sensemakers. We integrated rationale extraction models into SenseMate through an iterative, human-centered design process revolving around four key design principles derived from an extensive literature review. The design process consisted of three iterations with continuous feedback from seven people associated with community organizations. Through an online experiment involving 180 novice sensemakers, we aimed to determine whether AI-generated recommendations and rationales would decrease coding time, increase intercoder reliability (i.e. Cohen’s kappa), and minimize differences between novice and expert coding decisions (i.e. F-score of participant answers compared to expert gold labels). We found that though the model recommendations and explanations increased coding time by 49 seconds per unit of analysis, they raised intercoder reliability by 29% and coding F-score by 10%. Regarding the effectiveness of SenseMate’s design, participants reported that the platform was generally easy to use. In summary, Sensemate is (1) built for beginner sensemakers without a technical background, a user group that prior work doesn’t focus on, (2) implements rationale extraction models to recommend themes and generate explanations, which has advantages over large language models in terms of user privacy and control, and (3) contains original and intuitive features created from user feedback that can be applied to future QDA systems.
MIT Department
Massachusetts Institute of Technology. Media Laboratory
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1145/3640543.3645194
