AI-assisted analysis of content, structure, and sentiment in MOOC discussion forums
Name
feduc-08-1250846.pdf
Description
Published version
Size
1.24 MB
Format
Adobe PDF
Checksum (MD5)
f14ec9a28b34a9fa8565af413c0db7fa
Author(s)
Yee, Michael
Roy, Anindya
Perdue, Meghan
Cuevas, Consuelo
Quigley, Keegan
Bell, Ana
Rungta, Ahaan
Miyagawa, Shigeru
Date Issued
September 15, 2023
Journal
Frontiers in Education
Publisher
Frontiers Media SA
Citation
Yee M, Roy A, Perdue M, Cuevas C, Quigley K, Bell A, Rungta A and Miyagawa S (2023) AI-assisted analysis of content, structure, and sentiment in MOOC discussion forums. Front. Educ. 8:1250846.
Version
Final published version
Abstract
Discussion forums are a key component of online learning platforms, allowing learners to ask for help, provide help to others, and connect with others in the learning community. Analyzing patterns of forum usage and their association with course outcomes can provide valuable insight into how learners actually use discussion forums, and suggest strategies for shaping forum dynamics to improve learner experiences and outcomes. However, the fine-grained coding of forum posts required for this kind of analysis is a manually intensive process that can be challenging for large datasets, e.g., those that result from popular MOOCs. To address this issue, we propose an AI-assisted labeling process that uses advanced natural language processing techniques to train machine learning models capable of labeling a large dataset while minimizing human annotation effort. We fine-tune pretrained transformer-based deep learning models on category, structure, and emotion classification tasks. The transformer-based models outperform a more traditional baseline that uses support vector machines and a bag-of-words input representation. The transformer-based models also perform better when we augment the input features for an individual post with additional context from the post's thread (e.g., the thread title). We validate model quality through a combination of internal performance metrics, human auditing, and common-sense checks. For our Python MOOC dataset, we find that annotating approximately 1% of the forum posts achieves performance levels that are reliable for downstream analysis. Using labels from the validated AI models, we investigate the association of learner and course attributes with thread resolution and various forms of forum participation. We find significant differences in how learners of different age groups, gender, and course outcome status ask for help, provide help, and make posts with emotional (positive or negative) sentiment.
MIT Department
Massachusetts Institute of Technology. Artificial Intelligence Laboratory
Lincoln Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Massachusetts Institute of Technology. Department of Linguistics and Philosophy
Terms of Use
Creative Commons Attribution
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.3389/feduc.2023.1250846
