This is not the latest version of this item. The latest version can be found here.
Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction
Name
2020.conll-1.49.pdf
Description
Published version
Size
468.03 KB
Format
Adobe PDF
Checksum (MD5)
8114b5ed6feeab20595c1af88d5a7c10
Author(s)
Eisape, Tiwalayo
Zaslavsky, Noga
Levy, Roger
Date Issued
2020
Journal
Proceedings of the 24th Conference on Computational Natural Language Learning
Publisher
Association for Computational Linguistics (ACL)
Citation
Eisape, Tiwalayo, Zaslavsky, Noga and Levy, Roger. 2020. "Cloze Distillation: Improving Neural Language Models with Human Next-Word Prediction." Proceedings of the 24th Conference on Computational Natural Language Learning.
Version
Final published version
Abstract
Contemporary autoregressive language models (LMs) trained purely on corpus data have
been shown to capture numerous features of
human incremental processing. However, past
work has also suggested dissociations between
corpus probabilities and human next-word predictions. Here we evaluate several state-of-theart language models for their match to human
next-word predictions and to reading time behavior from eye movements. We then propose
a novel method for distilling the linguistic information implicit in human linguistic predictions into pre-trained LMs: Cloze Distillation.
We apply this method to a baseline neural LM
and show potential improvement in reading
time prediction and generalization to held-out
human cloze data.
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
10.18653/V1/2020.CONLL-1.49
