Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
Name
N19-1334.pdf
Description
Published version
Size
1.35 MB
Format
Unknown
Checksum (MD5)
84d1d0cb201544f742d07184253dfd4d
Author(s)
Wilcox, Ethan
Qian, Peng
Futrell, Richard
Ballesteros, Miguel
Levy, Roger P
Date Issued
June 2019
Journal
NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference
Publisher
Association for Computational Linguistics
Citation
2019. "Structural Supervision Improves Learning of Non-Local Grammatical Dependencies." NAACL HLT 2019 - 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 1.
Version
Final published version
Abstract
© 2019 Association for Computational Linguistics State-of-the-art LSTM language models trained on large corpora learn sequential contingencies in impressive detail and have been shown to acquire a number of non-local grammatical dependencies with some success. Here we investigate whether supervision with hierarchical structure enhances learning of a range of grammatical dependencies, a question that has previously been addressed only for subject-verb agreement. Using controlled experimental methods from psycholinguistics, we compare the performance of word-based LSTM models versus two models that represent hierarchical structure and deploy it in left-to-right processing: Recurrent Neural Network Grammars (RNNGs) (Dyer et al., 2016) and a incrementalized version of the Parsing-as-Language-Modeling configuration from Charniak et al. (2016). Models are tested on a diverse range of configurations for two classes of non-local grammatical dependencies in English-Negative Polarity licensing and Filler-Gap Dependencies. Using the same training data across models, we find that structurally-supervised models outperform the LSTM, with the RNNG demonstrating best results on both types of grammatical dependencies and even learning many of the Island Constraints on the filler-gap dependency. Structural supervision thus provides data efficiency advantages over purely string-based training of neural language models in acquiring human-like generalizations about non-local grammatical dependencies.
MIT Department
Massachusetts Institute of Technology. Department of Brain and Cognitive Sciences
MIT-IBM Watson AI Lab
Terms of Use
Article is made available in accordance with the publisher's policy and may be subject to US copyright law. Please refer to the publisher's site for terms of use.
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.18653/V1/N19-1334
