Essays on Using Large Language Models in Forecasting and Knowledge Augmentation
Name
Song-jaeyoons-Management-2026-thesis.pdf
Size
12.13 MB
Format
Adobe PDF
Checksum (MD5)
e4e3fef20b1eaa226638d49a10331369
Author(s)
Song, Jaeyoon
Advisor(s)
Malone, Thomas W.
Date Issued
May 2026
Publisher
Massachusetts Institute of Technology
Abstract
Large language models (LLMs) have advanced rapidly, raising new questions about how their capabilities are evaluated and how they interact with individuals and surrounding communities. This dissertation examines the usage of LLMs across three contexts. It begins by developing a framework for assessing models in isolation on event-based forecasting tasks. It then introduces humans into the loop, evaluating how the value of human-AI collaboration shifts as model capabilities improve. Finally, it moves beyond controlled settings to examine how the diffusion of LLMs correlates with patterns of participation in an online knowledge-sharing community.
Chapter 1 discusses the evaluation of LLMs on event-based forecasting. This chapter proposes a backtesting framework that evaluates forecasts of public events based only on knowledge that was available before the event occurred. The framework uses frozen context snapshots: forecasting questions paired with contemporaneous, structured summaries of web search results relevant to the questions. The pipeline collects unresolved questions from prediction markets and captures their supporting context at the time of collection, eliminating temporal contamination.
Chapter 2 examines how the value of human-AI collaboration changes as LLM systems become more capable. We provide an empirical instantiation through the case of short story writing. From the point of view of speed and quality, our experiment characterizes the progression in three stages: (a) AI provides limited benefit, (b) humans and AI achieve synergy, and (c) the marginal contribution of human input declines. The findings suggest that, as AI capabilities continue to advance, the nature of human-AI collaboration may be redefined, with the human role shifting from direct performance augmentation toward fostering exploration and conceptual diversity.
Chapter 3 turns to how the diffusion of LLMs correlates with patterns of contributor engagement on online knowledge communities. Through a descriptive, correlational analysis of Stack Overflow, we document patterns and shifts in contributor engagement following the release of ChatGPT. We cluster 394,295 users into canonical contributor profiles based on pre-ChatGPT behavioral data. Using a two-step method combining graph-based community detection and guided Latent Dirichlet Allocation, we identify nine user roles. A matched panel comparison with a difference-in-differences-style specification documents bounded heterogeneity across clusters within a context of platform-wide decline.
MIT Department
Sloan School of Management
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
