Anchoring and Agreement in Syntactic Annotations
Name
CBMM-Memo-055.pdf
Size
768.54 KB
Format
Adobe PDF
Checksum (MD5)
5e19c1d9cf1ae71944ab0d5c2af175d5
Author(s)
Berzak, Yevgeni
Huang, Yan
Barbu, Andrei
Korhonen, Anna
Katz, Boris
Date Issued
September 21, 2016
Publisher
Center for Brains, Minds and Machines (CBMM), arXiv
Citation
arXiv:1605.04481
Series/Report no.
CBMM Memo Series;055
Abstract
Published in the Proceedings of EMNLP 2016
We present a study on two key characteristics of human syntactic annotations: anchoring and agreement. Anchoring is a well-known cognitive bias in human decision making, where judgments are drawn towards preexisting values. We study the influence of anchoring on a standard approach to creation of syntactic resources where syntactic annotations are obtained via human editing of tagger and parser output. Our experiments demonstrate a clear anchoring effect and reveal unwanted consequences, including overestimation of parsing performance and lower quality of annotations in comparison with human-based annotations. Using sentences from the Penn Treebank WSJ, we also report systematically obtained inter-annotator agreement estimates for English dependency parsing. Our agreement results control for parser bias, and are consequential in that they are on par with state of the art parsing performance for English newswire. We discuss the impact of our findings on strategies for future annotation efforts and parser evaluations.
Subjects
human syntactic annotations
Anchoring
Agreement
Terms of Use
Attribution-NonCommercial-ShareAlike 3.0 United States
Persistent DSpace Link
