Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction
Name
2020.conll-1.49.pdf
Description
Published version
Size
468.03 KB
Format
Unknown
Checksum (MD5)
8114b5ed6feeab20595c1af88d5a7c10
Author(s)
Eisape, Tiwalayo
Zaslavsky, Noga
Levy, Roger
Date Issued
November 2020
Journal
Proceedings of the 24th Conference on Computational Natural Language Learning
Publisher
Association for Computational Linguistics (ACL)
Citation
Eisape, Tiwalayo, Zaslavsky, Noga and Levy, Roger. 2020. "Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction." Proceedings of the 24th Conference on Computational Natural Language Learning.
Version
Final published version
Abstract
Contemporary autoregressive language models (LMs) trained purely on corpus data have
been shown to capture numerous features of
human incremental processing. However, past
work has also suggested dissociations between
corpus probabilities and human next-word predictions. Here we evaluate several state-of-theart language models for their match to human
next-word predictions and to reading time behavior from eye movements. We then propose
a novel method for distilling the linguistic information implicit in human linguistic predictions into pre-trained LMs: Cloze Distillation.
We apply this method to a baseline neural LM
and show potential improvement in reading
time prediction and generalization to held-out
human cloze data.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Center for Brains, Minds, and Machines
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.18653/V1/2020.CONLL-1.49
