SenseMate: An AI-Based Platform to Support Qualitative Coding

Abstract

Unstructured data can be analyzed numerically or qualitatively through methods like sensemaking. One of the key stages of sensemaking is qualitative coding, where the data is divided into units, and each unit is assigned a category or code. Unfortunately, coding is tedious and time-consuming when carried out manually. Finding a balance between manual and fully-automated coding can help increase efficiency while allowing human judgment and preventing systematic machine errors. In this thesis, I propose an accessible semi-automated approach to qualitative coding. First, I apply a novel machine learning method, rationale extraction models, to qualitative coding. These models recommend themes for each unit of analysis in qualitative data and tend to perform better with less ambiguous themes. Through an online experiment, I find that assistance from rationale extraction models increases coding performance and reliability. Next, I execute an iterative, human-centered design process to create SenseMate, an AI-based platform for qualitative coding. After 13 user testing sessions and 3 design iterations, I observe that model overreliance can be minimized through cognitive forcing functions and easy-to-understand model explanations. I also design several ways for users to efficiently provide feedback on machine-generated rationales. To connect my model and design evaluations, I implement a prototype of SenseMate and conduct a summative user evaluation through an online experiment. The evaluation reveals that participants with access to AI assistance have higher coding performances but spend more time on the platform. The effectiveness of various design decisions within SenseMate is also explored. Finally, I discuss a myriad of future work possibilities. Overall, this thesis offers a practical and accessible solution to analyzing unstructured data, which has broad applications for researchers and organizations across various fields.