SenseMate: An AI-Based Platform to Support Qualitative Coding
Name
overney-coverney-MS-MAS-2023-thesis.pdf
Description
Thesis PDF
Size
12.08 MB
Format
Adobe PDF
Checksum (MD5)
cea90ef2ed297757662bdc65e2d9c862
Author(s)
Overney, Cassandra
Advisor(s)
Roy, Deb
Date Issued
June 2023
Publisher
Massachusetts Institute of Technology
Abstract
Unstructured data can be analyzed numerically or qualitatively through methods like sensemaking. One of the key stages of sensemaking is qualitative coding, where the data is divided into units, and each unit is assigned a category or code. Unfortunately, coding is tedious and time-consuming when carried out manually. Finding a balance between manual and fully-automated coding can help increase efficiency while allowing human judgment and preventing systematic machine errors. In this thesis, I propose an accessible semi-automated approach to qualitative coding. First, I apply a novel machine learning method, rationale extraction models, to qualitative coding. These models recommend themes for each unit of analysis in qualitative data and tend to perform better with less ambiguous themes. Through an online experiment, I find that assistance from rationale extraction models increases coding performance and reliability. Next, I execute an iterative, human-centered design process to create SenseMate, an AI-based platform for qualitative coding. After 13 user testing sessions and 3 design iterations, I observe that model overreliance can be minimized through cognitive forcing functions and easy-to-understand model explanations. I also design several ways for users to efficiently provide feedback on machine-generated rationales. To connect my model and design evaluations, I implement a prototype of SenseMate and conduct a summative user evaluation through an online experiment. The evaluation reveals that participants with access to AI assistance have higher coding performances but spend more time on the platform. The effectiveness of various design decisions within SenseMate is also explored. Finally, I discuss a myriad of future work possibilities. Overall, this thesis offers a practical and accessible solution to analyzing unstructured data, which has broad applications for researchers and organizations across various fields.
MIT Department
Program in Media Arts and Sciences (Massachusetts Institute of Technology)
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
