Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing
Name
N19-1162.pdf
Description
Published version
Size
3.84 MB
Format
Adobe PDF
Checksum (MD5)
c8173f23aa9f28d63e75deff5ce2670e
Author(s)
Schuster, Tal
Ram, Ori
Barzilay, Regina
Globerson, Amir
Date Issued
2019
Journal
2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Publisher
Association for Computational Linguistics
Citation
Schuster, Tal et al. "Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing." 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, June 2019, Minneapolis, Minnesota, Association for Computational Linguistics, 2019. © 2019 Association for Computational Linguistics
Version
Final published version
Abstract
We introduce a novel method for multilingual transfer that utilizes deep contextual embeddings, pretrained in an unsupervised fashion. While contextual embeddings have been shown to yield richer representations of meaning compared to their static counterparts, aligning them poses a challenge due to their dynamic nature. To this end, we construct context-independent variants of the original monolingual spaces and utilize their mapping to derive an alignment for the context-dependent spaces. This mapping readily supports processing of a target language, improving transfer by context-aware embeddings. Our experimental results demonstrate the effectiveness of this approach for zero-shot and few-shot learning of dependency parsing. Specifically, our method consistently outperforms the previous state-of-the-art on 6 tested languages, yielding an improvement of 6.8 LAS points on average.
MIT Department
Massachusetts Institute of Technology. Computer Science and Artificial Intelligence Laboratory
Massachusetts Institute of Technology. Department of Electrical Engineering and Computer Science
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
https://doi.org/10.18653/V1/N19-1162
