This is not the latest version of this item. The latest version can be found here.
SyntaxGym: An Online Platform for Targeted Evaluation of Language Models
Name
2020.acl-demos.10.pdf
Description
Published version
Size
633 KB
Format
Adobe PDF
Checksum (MD5)
f285270d52bc13107d8db12446c49bd7
Author(s)
Gauthier, Jon
Hu, Jennifer
Wilcox, Ethan
Qian, Peng
Levy, Roger
Date Issued
2020
Journal
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations
Publisher
Association for Computational Linguistics (ACL)
Citation
Gauthier, Jon, Hu, Jennifer, Wilcox, Ethan, Qian, Peng and Levy, Roger. 2020. "SyntaxGym: An Online Platform for Targeted Evaluation of Language Models." Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations.
Version
Final published version
Abstract
Targeted syntactic evaluations have yielded insights into the generalizations learned by neural network language models. However, this
line of research requires an uncommon confluence of skills: both the theoretical knowledge needed to design controlled psycholinguistic experiments, and the technical proficiency needed to train and deploy large-scale
language models. We present SyntaxGym,
an online platform designed to make targeted
evaluations accessible to both experts in NLP
and linguistics, reproducible across computing environments, and standardized following the norms of psycholinguistic experimental design. This paper releases two tools of independent value for the computational linguistics community:
1. A website, syntaxgym.org, which
centralizes the process of targeted syntactic evaluation and provides easy tools for
analysis and visualization;
2. Two command-line tools, syntaxgym
and lm-zoo, which allow any user to
reproduce targeted syntactic evaluations
and general language model inference on
their own machine.
Terms of Use
Creative Commons Attribution 4.0 International license
Persistent DSpace Link
DOI of Published Version
10.18653/V1/2020.ACL-DEMOS.10
