Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations
Name
W19-4819.pdf
Description
Published version
Size
843.43 KB
Format
Adobe PDF
Checksum (MD5)
0fca4f8dfc96f99b1bd2b084fda6190b
Author(s)
Wilcox, Ethan
Levy, Roger P
Futrell, Richard
Date Issued
August 2019
Journal
Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP
Publisher
Association for Computational Linguistics
Citation
Wilcox, Ethan et al. "Hierarchical Representation in Neural Language Models: Suppression and Recovery of Expectations." Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, August 2019, Florence, Italy, Association for Computational Linguistics, August 2019.
Version
Final published version
Abstract
Work using artificial languages as training input has shown that LSTMs are capable of inducing the stack-like data structures required to represent context-free and certain mildly context-sensitive languages — formal language classes which correspond in theory to the hierarchical structures of natural language. Here we present a suite of experiments probing whether neural language models trained on linguistic data induce these stack-like data structures and deploy them while incrementally predicting words. We study two natural language phenomena: center embedding sentences and syntactic island constraints on the filler–gap dependency. In order to properly predict words in these structures, a model must be able to temporarily suppress certain expectations and then recover those expectations later, essentially pushing and popping these expectations on a stack. Our results provide evidence that models can successfully suppress and recover expectations in many cases, but do not fully recover their previous grammatical state.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.18653/v1/w19-4819
