Explaining the Mechanisms of Compositionality of Small
Transformers
Name
sarma_tssarma_meng_bcs_thesis_2026.pdf
Size
10.43 MB
Format
Adobe PDF
Checksum (MD5)
72e889ddb23e65a59bfa99b3c295bc13
Author(s)
Sarma, Tara
Advisor(s)
Jazayeri, Mehrdad
Date Issued
February 2026
Publisher
Massachusetts Institute of Technology
Abstract
Compositionality is the principle of analyzing a complex expression through its subparts, or through a composition of smaller, simpler functions. In theories of learning, it is often considered a necessary component of generalization, as it is a much more flexible strategy than memorization. This has become an increasing concern in the study of artificial neural networks, which often generalize much worse than humans. Past methods that address this include Meta-Learning for Compositionality, a method where a transformer model is trained to answer compositional, symbolic input/output statements based on a set of examples. While this method is behaviorally successful, no insight is given into how this compositional behavior arises neurally. Thus, I unpack the neural mechanisms of a transformer model trained with this method. Using these results, I provide a theory for the neural implementation of compositionality in small transformers.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
In Copyright - Educational Use Permitted
Copyright retained by author(s)
Persistent DSpace Link
