Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training
Name
nol_a_00137.pdf
Size
1018.42 KB
Format
Adobe PDF
Checksum (MD5)
df89a94ddf3e5cdcd64922b756c41faf
Author(s)
Hosseini, Eghbal A.
Schrimpf, Martin
Zhang, Yian
Bowman, Samuel
Zaslavsky, Noga
Fedorenko, Evelina
Date Issued
2024
Journal
Neurobiology of Language
Publisher
MIT Press
Citation
Eghbal A. Hosseini, Martin Schrimpf, Yian Zhang, Samuel Bowman, Noga Zaslavsky, Evelina Fedorenko; Artificial Neural Network Language Models Predict Human Brain Responses to Language Even After a Developmentally Realistic Amount of Training. Neurobiology of Language 2024; 5 (1): 43–63.
Version
Final published version
Abstract
Artificial neural networks have emerged as computationally plausible models of human language processing. A major criticism of these models is that the amount of training data they receive far exceeds that of humans during language learning. Here, we use two complementary approaches to ask how the models’ ability to capture human fMRI responses to sentences is affected by the amount of training data. First, we evaluate GPT-2 models trained on 1 million, 10 million, 100 million, or 1 billion words against an fMRI benchmark. We consider the 100-million-word model to be developmentally plausible in terms of the amount of training data given that this amount is similar to what children are estimated to be exposed to during the first 10 years of life. Second, we test the performance of a GPT-2 model trained on a 9-billion-token dataset to reach state-of-the-art next-word prediction performance on the human benchmark at different stages during training. Across both approaches, we find that (i) the models trained on a developmentally plausible amount of data already achieve near-maximal performance in capturing fMRI responses to sentences. Further, (ii) lower perplexity—a measure of next-word prediction performance—is associated with stronger alignment with human data, suggesting that models that have received enough training to achieve sufficiently high next-word prediction performance also acquire representations of sentences that are predictive of human fMRI responses. In tandem, these findings establish that although some training is necessary for the models’ predictive ability, a developmentally realistic amount of training (∼100 million words) may suffice.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
McGovern Institute for Brain Research at MIT
MIT Quest for Intelligence
Terms of Use
Creative Commons Attribution
An error occurred on the license name.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.1162/nol_a_00137
